Newspaper library
As an absolute newbie on the topic of using python, I stumbled over a few difficulties using the newspaper library extension. My goal is to use the newspaper extension on a regular basis to download all new articles of a German news website called "tagesschau" and all articles from CNN to build a data stack I can analyze in a few years.
If I got it right I could use the following commands to download and scrape all articles into the python library.
import newspaper
from newspaper import news_pool
tagesschau_paper = newspaper.build('http://tagesschau.de')
cnn_paper = newspaper.build('http://cnn.com')
papers = [tagesschau_paper, cnn_paper]
news_pool.set(papers, threads_per_source=2) # (3*2) = 6 threads total
news_pool.join()`
If that's the right way to download all articles, so how I can extract and save those outside of python? Or saving those articles in python so that I can reuse them if I restart python again?
Thanks for your help.
python python-newspaper
add a comment |
As an absolute newbie on the topic of using python, I stumbled over a few difficulties using the newspaper library extension. My goal is to use the newspaper extension on a regular basis to download all new articles of a German news website called "tagesschau" and all articles from CNN to build a data stack I can analyze in a few years.
If I got it right I could use the following commands to download and scrape all articles into the python library.
import newspaper
from newspaper import news_pool
tagesschau_paper = newspaper.build('http://tagesschau.de')
cnn_paper = newspaper.build('http://cnn.com')
papers = [tagesschau_paper, cnn_paper]
news_pool.set(papers, threads_per_source=2) # (3*2) = 6 threads total
news_pool.join()`
If that's the right way to download all articles, so how I can extract and save those outside of python? Or saving those articles in python so that I can reuse them if I restart python again?
Thanks for your help.
python python-newspaper
add a comment |
As an absolute newbie on the topic of using python, I stumbled over a few difficulties using the newspaper library extension. My goal is to use the newspaper extension on a regular basis to download all new articles of a German news website called "tagesschau" and all articles from CNN to build a data stack I can analyze in a few years.
If I got it right I could use the following commands to download and scrape all articles into the python library.
import newspaper
from newspaper import news_pool
tagesschau_paper = newspaper.build('http://tagesschau.de')
cnn_paper = newspaper.build('http://cnn.com')
papers = [tagesschau_paper, cnn_paper]
news_pool.set(papers, threads_per_source=2) # (3*2) = 6 threads total
news_pool.join()`
If that's the right way to download all articles, so how I can extract and save those outside of python? Or saving those articles in python so that I can reuse them if I restart python again?
Thanks for your help.
python python-newspaper
As an absolute newbie on the topic of using python, I stumbled over a few difficulties using the newspaper library extension. My goal is to use the newspaper extension on a regular basis to download all new articles of a German news website called "tagesschau" and all articles from CNN to build a data stack I can analyze in a few years.
If I got it right I could use the following commands to download and scrape all articles into the python library.
import newspaper
from newspaper import news_pool
tagesschau_paper = newspaper.build('http://tagesschau.de')
cnn_paper = newspaper.build('http://cnn.com')
papers = [tagesschau_paper, cnn_paper]
news_pool.set(papers, threads_per_source=2) # (3*2) = 6 threads total
news_pool.join()`
If that's the right way to download all articles, so how I can extract and save those outside of python? Or saving those articles in python so that I can reuse them if I restart python again?
Thanks for your help.
python python-newspaper
python python-newspaper
edited Nov 16 at 9:23
SiHa
3,24961632
3,24961632
asked Nov 13 at 21:02
Philipp Schulz
132
132
add a comment |
add a comment |
2 Answers
2
active
oldest
votes
The following codes will save the downloaded articles in HTML format. In the folder, you'll find. tagesschau_paper0.html, tagesschau_paper1.html, tagesschau_paper2.html, .....
import newspaper
from newspaper import news_pool
tagesschau_paper = newspaper.build('http://tagesschau.de')
cnn_paper = newspaper.build('http://cnn.com')
papers = [tagesschau_paper, cnn_paper]
news_pool.set(papers, threads_per_source=2)
news_pool.join()
for i in range (tagesschau_paper.size()):
with open("tagesschau_paper{}.html".format(i), "w") as file:
file.write(tagesschau_paper.articles[i].html)
Note: news_pool
doesn't get anything from CNN, so I skipped to write codes for it. If you check cnn_paper.size()
, it results to 0
. You have to import and use Source instead.
The above codes can be followed as an example to save articles in other formats too, e.g. txt and also only parts that you need from the articles e.g. authors, body, publish_date.
Thanks for youre help :)
– Philipp Schulz
Nov 16 at 11:19
I had to adapt the code to support UTF-8 Encoding: 'with open("tagesschau_paper{}.html".format(i), "w" ,encoding='utf-8') as file:`
– Philipp Schulz
Nov 18 at 11:53
Thanks for the update. I didn't need to set the encoding.
– Mrinal Roy
Nov 21 at 2:20
add a comment |
You can use pickle to save objects outside of python and reopen them later:
file_Name = "testfile"
# open the file for writing
fileObject = open(file_Name,'wb')
# this writes the object news_pool to the
# file named 'testfile'
pickle.dump(news_pool,fileObject)
# here we close the fileObject
fileObject.close()
# we open the file for reading
fileObject = open(file_Name,'r')
# load the object from the file into var news_pool_reopen
news_pool_reopen = pickle.load(fileObject)
Using pickle with custom objects is dangerous: during unpacking, pickle instantiates an instance of the encountered class and then populates its attributes with the data from the file. If the class implementation ofnews_pool
changes (which seems likely given the time span OP specified and the fact thatnewspaper
seems to be in active development), thenpickle.load
will likely fail. See, for example, this explanation.
– Paul Brodersen
Nov 13 at 23:07
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53289440%2fnewspaper-library%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
2 Answers
2
active
oldest
votes
2 Answers
2
active
oldest
votes
active
oldest
votes
active
oldest
votes
The following codes will save the downloaded articles in HTML format. In the folder, you'll find. tagesschau_paper0.html, tagesschau_paper1.html, tagesschau_paper2.html, .....
import newspaper
from newspaper import news_pool
tagesschau_paper = newspaper.build('http://tagesschau.de')
cnn_paper = newspaper.build('http://cnn.com')
papers = [tagesschau_paper, cnn_paper]
news_pool.set(papers, threads_per_source=2)
news_pool.join()
for i in range (tagesschau_paper.size()):
with open("tagesschau_paper{}.html".format(i), "w") as file:
file.write(tagesschau_paper.articles[i].html)
Note: news_pool
doesn't get anything from CNN, so I skipped to write codes for it. If you check cnn_paper.size()
, it results to 0
. You have to import and use Source instead.
The above codes can be followed as an example to save articles in other formats too, e.g. txt and also only parts that you need from the articles e.g. authors, body, publish_date.
Thanks for youre help :)
– Philipp Schulz
Nov 16 at 11:19
I had to adapt the code to support UTF-8 Encoding: 'with open("tagesschau_paper{}.html".format(i), "w" ,encoding='utf-8') as file:`
– Philipp Schulz
Nov 18 at 11:53
Thanks for the update. I didn't need to set the encoding.
– Mrinal Roy
Nov 21 at 2:20
add a comment |
The following codes will save the downloaded articles in HTML format. In the folder, you'll find. tagesschau_paper0.html, tagesschau_paper1.html, tagesschau_paper2.html, .....
import newspaper
from newspaper import news_pool
tagesschau_paper = newspaper.build('http://tagesschau.de')
cnn_paper = newspaper.build('http://cnn.com')
papers = [tagesschau_paper, cnn_paper]
news_pool.set(papers, threads_per_source=2)
news_pool.join()
for i in range (tagesschau_paper.size()):
with open("tagesschau_paper{}.html".format(i), "w") as file:
file.write(tagesschau_paper.articles[i].html)
Note: news_pool
doesn't get anything from CNN, so I skipped to write codes for it. If you check cnn_paper.size()
, it results to 0
. You have to import and use Source instead.
The above codes can be followed as an example to save articles in other formats too, e.g. txt and also only parts that you need from the articles e.g. authors, body, publish_date.
Thanks for youre help :)
– Philipp Schulz
Nov 16 at 11:19
I had to adapt the code to support UTF-8 Encoding: 'with open("tagesschau_paper{}.html".format(i), "w" ,encoding='utf-8') as file:`
– Philipp Schulz
Nov 18 at 11:53
Thanks for the update. I didn't need to set the encoding.
– Mrinal Roy
Nov 21 at 2:20
add a comment |
The following codes will save the downloaded articles in HTML format. In the folder, you'll find. tagesschau_paper0.html, tagesschau_paper1.html, tagesschau_paper2.html, .....
import newspaper
from newspaper import news_pool
tagesschau_paper = newspaper.build('http://tagesschau.de')
cnn_paper = newspaper.build('http://cnn.com')
papers = [tagesschau_paper, cnn_paper]
news_pool.set(papers, threads_per_source=2)
news_pool.join()
for i in range (tagesschau_paper.size()):
with open("tagesschau_paper{}.html".format(i), "w") as file:
file.write(tagesschau_paper.articles[i].html)
Note: news_pool
doesn't get anything from CNN, so I skipped to write codes for it. If you check cnn_paper.size()
, it results to 0
. You have to import and use Source instead.
The above codes can be followed as an example to save articles in other formats too, e.g. txt and also only parts that you need from the articles e.g. authors, body, publish_date.
The following codes will save the downloaded articles in HTML format. In the folder, you'll find. tagesschau_paper0.html, tagesschau_paper1.html, tagesschau_paper2.html, .....
import newspaper
from newspaper import news_pool
tagesschau_paper = newspaper.build('http://tagesschau.de')
cnn_paper = newspaper.build('http://cnn.com')
papers = [tagesschau_paper, cnn_paper]
news_pool.set(papers, threads_per_source=2)
news_pool.join()
for i in range (tagesschau_paper.size()):
with open("tagesschau_paper{}.html".format(i), "w") as file:
file.write(tagesschau_paper.articles[i].html)
Note: news_pool
doesn't get anything from CNN, so I skipped to write codes for it. If you check cnn_paper.size()
, it results to 0
. You have to import and use Source instead.
The above codes can be followed as an example to save articles in other formats too, e.g. txt and also only parts that you need from the articles e.g. authors, body, publish_date.
edited Nov 16 at 8:57
answered Nov 16 at 8:49
Mrinal Roy
713
713
Thanks for youre help :)
– Philipp Schulz
Nov 16 at 11:19
I had to adapt the code to support UTF-8 Encoding: 'with open("tagesschau_paper{}.html".format(i), "w" ,encoding='utf-8') as file:`
– Philipp Schulz
Nov 18 at 11:53
Thanks for the update. I didn't need to set the encoding.
– Mrinal Roy
Nov 21 at 2:20
add a comment |
Thanks for youre help :)
– Philipp Schulz
Nov 16 at 11:19
I had to adapt the code to support UTF-8 Encoding: 'with open("tagesschau_paper{}.html".format(i), "w" ,encoding='utf-8') as file:`
– Philipp Schulz
Nov 18 at 11:53
Thanks for the update. I didn't need to set the encoding.
– Mrinal Roy
Nov 21 at 2:20
Thanks for youre help :)
– Philipp Schulz
Nov 16 at 11:19
Thanks for youre help :)
– Philipp Schulz
Nov 16 at 11:19
I had to adapt the code to support UTF-8 Encoding: 'with open("tagesschau_paper{}.html".format(i), "w" ,encoding='utf-8') as file:`
– Philipp Schulz
Nov 18 at 11:53
I had to adapt the code to support UTF-8 Encoding: 'with open("tagesschau_paper{}.html".format(i), "w" ,encoding='utf-8') as file:`
– Philipp Schulz
Nov 18 at 11:53
Thanks for the update. I didn't need to set the encoding.
– Mrinal Roy
Nov 21 at 2:20
Thanks for the update. I didn't need to set the encoding.
– Mrinal Roy
Nov 21 at 2:20
add a comment |
You can use pickle to save objects outside of python and reopen them later:
file_Name = "testfile"
# open the file for writing
fileObject = open(file_Name,'wb')
# this writes the object news_pool to the
# file named 'testfile'
pickle.dump(news_pool,fileObject)
# here we close the fileObject
fileObject.close()
# we open the file for reading
fileObject = open(file_Name,'r')
# load the object from the file into var news_pool_reopen
news_pool_reopen = pickle.load(fileObject)
Using pickle with custom objects is dangerous: during unpacking, pickle instantiates an instance of the encountered class and then populates its attributes with the data from the file. If the class implementation ofnews_pool
changes (which seems likely given the time span OP specified and the fact thatnewspaper
seems to be in active development), thenpickle.load
will likely fail. See, for example, this explanation.
– Paul Brodersen
Nov 13 at 23:07
add a comment |
You can use pickle to save objects outside of python and reopen them later:
file_Name = "testfile"
# open the file for writing
fileObject = open(file_Name,'wb')
# this writes the object news_pool to the
# file named 'testfile'
pickle.dump(news_pool,fileObject)
# here we close the fileObject
fileObject.close()
# we open the file for reading
fileObject = open(file_Name,'r')
# load the object from the file into var news_pool_reopen
news_pool_reopen = pickle.load(fileObject)
Using pickle with custom objects is dangerous: during unpacking, pickle instantiates an instance of the encountered class and then populates its attributes with the data from the file. If the class implementation ofnews_pool
changes (which seems likely given the time span OP specified and the fact thatnewspaper
seems to be in active development), thenpickle.load
will likely fail. See, for example, this explanation.
– Paul Brodersen
Nov 13 at 23:07
add a comment |
You can use pickle to save objects outside of python and reopen them later:
file_Name = "testfile"
# open the file for writing
fileObject = open(file_Name,'wb')
# this writes the object news_pool to the
# file named 'testfile'
pickle.dump(news_pool,fileObject)
# here we close the fileObject
fileObject.close()
# we open the file for reading
fileObject = open(file_Name,'r')
# load the object from the file into var news_pool_reopen
news_pool_reopen = pickle.load(fileObject)
You can use pickle to save objects outside of python and reopen them later:
file_Name = "testfile"
# open the file for writing
fileObject = open(file_Name,'wb')
# this writes the object news_pool to the
# file named 'testfile'
pickle.dump(news_pool,fileObject)
# here we close the fileObject
fileObject.close()
# we open the file for reading
fileObject = open(file_Name,'r')
# load the object from the file into var news_pool_reopen
news_pool_reopen = pickle.load(fileObject)
answered Nov 13 at 22:41
Turtalicious
524
524
Using pickle with custom objects is dangerous: during unpacking, pickle instantiates an instance of the encountered class and then populates its attributes with the data from the file. If the class implementation ofnews_pool
changes (which seems likely given the time span OP specified and the fact thatnewspaper
seems to be in active development), thenpickle.load
will likely fail. See, for example, this explanation.
– Paul Brodersen
Nov 13 at 23:07
add a comment |
Using pickle with custom objects is dangerous: during unpacking, pickle instantiates an instance of the encountered class and then populates its attributes with the data from the file. If the class implementation ofnews_pool
changes (which seems likely given the time span OP specified and the fact thatnewspaper
seems to be in active development), thenpickle.load
will likely fail. See, for example, this explanation.
– Paul Brodersen
Nov 13 at 23:07
Using pickle with custom objects is dangerous: during unpacking, pickle instantiates an instance of the encountered class and then populates its attributes with the data from the file. If the class implementation of
news_pool
changes (which seems likely given the time span OP specified and the fact that newspaper
seems to be in active development), then pickle.load
will likely fail. See, for example, this explanation.– Paul Brodersen
Nov 13 at 23:07
Using pickle with custom objects is dangerous: during unpacking, pickle instantiates an instance of the encountered class and then populates its attributes with the data from the file. If the class implementation of
news_pool
changes (which seems likely given the time span OP specified and the fact that newspaper
seems to be in active development), then pickle.load
will likely fail. See, for example, this explanation.– Paul Brodersen
Nov 13 at 23:07
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Some of your past answers have not been well-received, and you're in danger of being blocked from answering.
Please pay close attention to the following guidance:
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53289440%2fnewspaper-library%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown