Large XML Response-Python
0
I'm scraping a webpage that returns an XML response that I cannot for the life of me extract any data from. Here is my code that just returns the XML response:
import requests
url = 'https://www5.fdic.gov/cra/WebServices/DBService.asmx/callWS'
r = requests.post(url, data={"functionName":"SearchCRA","parmsJSON":"{"Appl_Number":"","Appl_Type":"","PSTALP":"","SUPRV_FDICDBS":"09","BANK_NAME":""}"})
print(r.content)
For example I would like to extract application numbers, institution names, and application type. I'm relatively new to Python and I just can't get my head around this one.
Thanks in advance.
python xml web-scraping
edited Nov 20 '18 at 16:42
SuperStew
1,6641717
asked Nov 20 '18 at 16:29
RRUDARYRRUDARY
267
add a comment |
0
I'm scraping a webpage that returns an XML response that I cannot for the life of me extract any data from. Here is my code that just returns the XML response:
import requests
url = 'https://www5.fdic.gov/cra/WebServices/DBService.asmx/callWS'
r = requests.post(url, data={"functionName":"SearchCRA","parmsJSON":"{"Appl_Number":"","Appl_Type":"","PSTALP":"","SUPRV_FDICDBS":"09","BANK_NAME":""}"})
print(r.content)
For example I would like to extract application numbers, institution names, and application type. I'm relatively new to Python and I just can't get my head around this one.
Thanks in advance.
python xml web-scraping
edited Nov 20 '18 at 16:42
SuperStew
1,6641717
asked Nov 20 '18 at 16:29
RRUDARYRRUDARY
267
have you looked into the xml libraries? That should help a lot
– SuperStew
Nov 20 '18 at 16:31
I cannot for the life of me extract any data from What have you tried?
– John Gordon
Nov 20 '18 at 16:34
I've tried using element tree and requests-xml and I haven't had much luck.
– RRUDARY
Nov 20 '18 at 17:19
add a comment |
0
0
0
I'm scraping a webpage that returns an XML response that I cannot for the life of me extract any data from. Here is my code that just returns the XML response:
import requests
url = 'https://www5.fdic.gov/cra/WebServices/DBService.asmx/callWS'
r = requests.post(url, data={"functionName":"SearchCRA","parmsJSON":"{"Appl_Number":"","Appl_Type":"","PSTALP":"","SUPRV_FDICDBS":"09","BANK_NAME":""}"})
print(r.content)
For example I would like to extract application numbers, institution names, and application type. I'm relatively new to Python and I just can't get my head around this one.
Thanks in advance.
python xml web-scraping
edited Nov 20 '18 at 16:42
SuperStew
1,6641717
asked Nov 20 '18 at 16:29
RRUDARYRRUDARY
267
I'm scraping a webpage that returns an XML response that I cannot for the life of me extract any data from. Here is my code that just returns the XML response:
import requests
url = 'https://www5.fdic.gov/cra/WebServices/DBService.asmx/callWS'
r = requests.post(url, data={"functionName":"SearchCRA","parmsJSON":"{"Appl_Number":"","Appl_Type":"","PSTALP":"","SUPRV_FDICDBS":"09","BANK_NAME":""}"})
print(r.content)
For example I would like to extract application numbers, institution names, and application type. I'm relatively new to Python and I just can't get my head around this one.
Thanks in advance.
python xml web-scraping
python xml web-scraping
edited Nov 20 '18 at 16:42
SuperStew
1,6641717
asked Nov 20 '18 at 16:29
RRUDARYRRUDARY
267
edited Nov 20 '18 at 16:42
SuperStew
1,6641717
asked Nov 20 '18 at 16:29
RRUDARYRRUDARY
267
edited Nov 20 '18 at 16:42
SuperStew
1,6641717
edited Nov 20 '18 at 16:42
SuperStew
1,6641717
edited Nov 20 '18 at 16:42
SuperStew
1,6641717
1,6641717
asked Nov 20 '18 at 16:29
RRUDARYRRUDARY
267
asked Nov 20 '18 at 16:29
RRUDARYRRUDARY
267
asked Nov 20 '18 at 16:29
RRUDARYRRUDARY
267
267
have you looked into the xml libraries? That should help a lot
– SuperStew
Nov 20 '18 at 16:31
I cannot for the life of me extract any data from What have you tried?
– John Gordon
Nov 20 '18 at 16:34
I've tried using element tree and requests-xml and I haven't had much luck.
– RRUDARY
Nov 20 '18 at 17:19
add a comment |
have you looked into the xml libraries? That should help a lot
– SuperStew
Nov 20 '18 at 16:31
I cannot for the life of me extract any data from What have you tried?
– John Gordon
Nov 20 '18 at 16:34
I've tried using element tree and requests-xml and I haven't had much luck.
– RRUDARY
Nov 20 '18 at 17:19
have you looked into the xml libraries? That should help a lot
– SuperStew
Nov 20 '18 at 16:31
have you looked into the xml libraries? That should help a lot
– SuperStew
Nov 20 '18 at 16:31
I cannot for the life of me extract any data from What have you tried?
– John Gordon
Nov 20 '18 at 16:34
I cannot for the life of me extract any data from What have you tried?
– John Gordon
Nov 20 '18 at 16:34
I've tried using element tree and requests-xml and I haven't had much luck.
– RRUDARY
Nov 20 '18 at 17:19
I've tried using element tree and requests-xml and I haven't had much luck.
– RRUDARY
Nov 20 '18 at 17:19
add a comment |
1 Answer
1
active
oldest
votes
0
The XML response actually has a very simple structure, with just a single root element <string>. The text of that element contains JSON, so actually parsing the content is trivial.
Assuming you have the response in r, then:
import json
from xml.etree import ElementTree as ET
root = ET.fromstring(r.content)
data = json.loads(root.text)
for result in data['Result']:
print(result['Appl_Number'])
print(result['Instname'])
print(result['Appl_Type'])
print('--')
answered Nov 21 '18 at 12:37
Will KeelingWill Keeling
12.2k22635
You are amazing, thank you so much. Would you mind explaining why the 'root' element is necessary? I'm not that familiar with any of this and I would love to learn more about it. Thanks again!
– RRUDARY
Nov 21 '18 at 13:57
The content of the response is just XML text. So we useET.fromstring()to parse that text and turn it into an in-memory XML representation.ET.fromstring()hands back the top level element (root element) of that in-memory XML representation (anElementinstance representing the<string>element). Once we have that, we can easily access that element's own nested text - which turns out is in JSON format. Since parsing JSON is trivial with Python'sjsonlibrary, it's downhill from then on.
– Will Keeling
Nov 21 '18 at 14:03
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
draft saved
draft discarded
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53397389%2flarge-xml-response-python%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
0
The XML response actually has a very simple structure, with just a single root element <string>. The text of that element contains JSON, so actually parsing the content is trivial.
Assuming you have the response in r, then:
import json
from xml.etree import ElementTree as ET
root = ET.fromstring(r.content)
data = json.loads(root.text)
for result in data['Result']:
print(result['Appl_Number'])
print(result['Instname'])
print(result['Appl_Type'])
print('--')
answered Nov 21 '18 at 12:37
Will KeelingWill Keeling
12.2k22635
You are amazing, thank you so much. Would you mind explaining why the 'root' element is necessary? I'm not that familiar with any of this and I would love to learn more about it. Thanks again!
– RRUDARY
Nov 21 '18 at 13:57
The content of the response is just XML text. So we useET.fromstring()to parse that text and turn it into an in-memory XML representation.ET.fromstring()hands back the top level element (root element) of that in-memory XML representation (anElementinstance representing the<string>element). Once we have that, we can easily access that element's own nested text - which turns out is in JSON format. Since parsing JSON is trivial with Python'sjsonlibrary, it's downhill from then on.
– Will Keeling
Nov 21 '18 at 14:03
add a comment |
0
The XML response actually has a very simple structure, with just a single root element <string>. The text of that element contains JSON, so actually parsing the content is trivial.
Assuming you have the response in r, then:
import json
from xml.etree import ElementTree as ET
root = ET.fromstring(r.content)
data = json.loads(root.text)
for result in data['Result']:
print(result['Appl_Number'])
print(result['Instname'])
print(result['Appl_Type'])
print('--')
answered Nov 21 '18 at 12:37
Will KeelingWill Keeling
12.2k22635
You are amazing, thank you so much. Would you mind explaining why the 'root' element is necessary? I'm not that familiar with any of this and I would love to learn more about it. Thanks again!
– RRUDARY
Nov 21 '18 at 13:57
The content of the response is just XML text. So we useET.fromstring()to parse that text and turn it into an in-memory XML representation.ET.fromstring()hands back the top level element (root element) of that in-memory XML representation (anElementinstance representing the<string>element). Once we have that, we can easily access that element's own nested text - which turns out is in JSON format. Since parsing JSON is trivial with Python'sjsonlibrary, it's downhill from then on.
– Will Keeling
Nov 21 '18 at 14:03
add a comment |
0
0
0
The XML response actually has a very simple structure, with just a single root element <string>. The text of that element contains JSON, so actually parsing the content is trivial.
Assuming you have the response in r, then:
import json
from xml.etree import ElementTree as ET
root = ET.fromstring(r.content)
data = json.loads(root.text)
for result in data['Result']:
print(result['Appl_Number'])
print(result['Instname'])
print(result['Appl_Type'])
print('--')
answered Nov 21 '18 at 12:37
Will KeelingWill Keeling
12.2k22635
The XML response actually has a very simple structure, with just a single root element <string>. The text of that element contains JSON, so actually parsing the content is trivial.
Assuming you have the response in r, then:
import json
from xml.etree import ElementTree as ET
root = ET.fromstring(r.content)
data = json.loads(root.text)
for result in data['Result']:
print(result['Appl_Number'])
print(result['Instname'])
print(result['Appl_Type'])
print('--')
answered Nov 21 '18 at 12:37
Will KeelingWill Keeling
12.2k22635
answered Nov 21 '18 at 12:37
Will KeelingWill Keeling
12.2k22635
answered Nov 21 '18 at 12:37
Will KeelingWill Keeling
12.2k22635
answered Nov 21 '18 at 12:37
Will KeelingWill Keeling
12.2k22635
12.2k22635
You are amazing, thank you so much. Would you mind explaining why the 'root' element is necessary? I'm not that familiar with any of this and I would love to learn more about it. Thanks again!
– RRUDARY
Nov 21 '18 at 13:57
The content of the response is just XML text. So we useET.fromstring()to parse that text and turn it into an in-memory XML representation.ET.fromstring()hands back the top level element (root element) of that in-memory XML representation (anElementinstance representing the<string>element). Once we have that, we can easily access that element's own nested text - which turns out is in JSON format. Since parsing JSON is trivial with Python'sjsonlibrary, it's downhill from then on.
– Will Keeling
Nov 21 '18 at 14:03
add a comment |
You are amazing, thank you so much. Would you mind explaining why the 'root' element is necessary? I'm not that familiar with any of this and I would love to learn more about it. Thanks again!
– RRUDARY
Nov 21 '18 at 13:57
The content of the response is just XML text. So we useET.fromstring()to parse that text and turn it into an in-memory XML representation.ET.fromstring()hands back the top level element (root element) of that in-memory XML representation (anElementinstance representing the<string>element). Once we have that, we can easily access that element's own nested text - which turns out is in JSON format. Since parsing JSON is trivial with Python'sjsonlibrary, it's downhill from then on.
– Will Keeling
Nov 21 '18 at 14:03
You are amazing, thank you so much. Would you mind explaining why the 'root' element is necessary? I'm not that familiar with any of this and I would love to learn more about it. Thanks again!
– RRUDARY
Nov 21 '18 at 13:57
You are amazing, thank you so much. Would you mind explaining why the 'root' element is necessary? I'm not that familiar with any of this and I would love to learn more about it. Thanks again!
– RRUDARY
Nov 21 '18 at 13:57
The content of the response is just XML text. So we use
ET.fromstring() to parse that text and turn it into an in-memory XML representation. ET.fromstring() hands back the top level element (root element) of that in-memory XML representation (an Element instance representing the <string> element). Once we have that, we can easily access that element's own nested text - which turns out is in JSON format. Since parsing JSON is trivial with Python's json library, it's downhill from then on.– Will Keeling
Nov 21 '18 at 14:03
The content of the response is just XML text. So we use
ET.fromstring() to parse that text and turn it into an in-memory XML representation. ET.fromstring() hands back the top level element (root element) of that in-memory XML representation (an Element instance representing the <string> element). Once we have that, we can easily access that element's own nested text - which turns out is in JSON format. Since parsing JSON is trivial with Python's json library, it's downhill from then on.– Will Keeling
Nov 21 '18 at 14:03
add a comment |
draft saved
draft discarded
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
draft saved
draft discarded
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53397389%2flarge-xml-response-python%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
have you looked into the xml libraries? That should help a lot
– SuperStew
Nov 20 '18 at 16:31
I cannot for the life of me extract any data from What have you tried?
– John Gordon
Nov 20 '18 at 16:34
I've tried using element tree and requests-xml and I haven't had much luck.
– RRUDARY
Nov 20 '18 at 17:19