How can I convert csv to RDD of labeled points?
I want to perform a decision tree regression on a dataset in csv. It is required that I do it using RDD. I tried the following to convert the dataframe to RDD:
pp_df = spark.read.csv("/usr/local/spark/data/hour.csv",header=True,inferSchema=True)
pp_df = pp_df.rdd.map(lambda x: LabeledPoint(x[10], x[:10])).collect()
Then I try to split the data for training and testing :
(trainingData, testData) = pp_df.randomSplit([0.7, 0.3])
And I got the following error:
AttributeError: 'list' object has no attribute 'randomSplit'
Why it returns a list instead and How can I properly convert the csv to RDD labeled data ?
python csv pyspark rdd
asked Nov 15 '18 at 9:58
FaureFaure
12
add a comment |
I want to perform a decision tree regression on a dataset in csv. It is required that I do it using RDD. I tried the following to convert the dataframe to RDD:
pp_df = spark.read.csv("/usr/local/spark/data/hour.csv",header=True,inferSchema=True)
pp_df = pp_df.rdd.map(lambda x: LabeledPoint(x[10], x[:10])).collect()
Then I try to split the data for training and testing :
(trainingData, testData) = pp_df.randomSplit([0.7, 0.3])
And I got the following error:
AttributeError: 'list' object has no attribute 'randomSplit'
Why it returns a list instead and How can I properly convert the csv to RDD labeled data ?
python csv pyspark rdd
asked Nov 15 '18 at 9:58
FaureFaure
12
in the title you mentioned RDD (resilient distributed dataset) and later you are mentioning RRD (round robin database).. I know you probably meant RDD because of spark usage, but please be consistent..
– Dejan Marić
Nov 15 '18 at 10:22
add a comment |
I want to perform a decision tree regression on a dataset in csv. It is required that I do it using RDD. I tried the following to convert the dataframe to RDD:
pp_df = spark.read.csv("/usr/local/spark/data/hour.csv",header=True,inferSchema=True)
pp_df = pp_df.rdd.map(lambda x: LabeledPoint(x[10], x[:10])).collect()
Then I try to split the data for training and testing :
(trainingData, testData) = pp_df.randomSplit([0.7, 0.3])
And I got the following error:
AttributeError: 'list' object has no attribute 'randomSplit'
Why it returns a list instead and How can I properly convert the csv to RDD labeled data ?
python csv pyspark rdd
asked Nov 15 '18 at 9:58
FaureFaure
12
I want to perform a decision tree regression on a dataset in csv. It is required that I do it using RDD. I tried the following to convert the dataframe to RDD:
pp_df = spark.read.csv("/usr/local/spark/data/hour.csv",header=True,inferSchema=True)
pp_df = pp_df.rdd.map(lambda x: LabeledPoint(x[10], x[:10])).collect()
Then I try to split the data for training and testing :
(trainingData, testData) = pp_df.randomSplit([0.7, 0.3])
And I got the following error:
AttributeError: 'list' object has no attribute 'randomSplit'
Why it returns a list instead and How can I properly convert the csv to RDD labeled data ?
python csv pyspark rdd
python csv pyspark rdd
asked Nov 15 '18 at 9:58
FaureFaure
12
asked Nov 15 '18 at 9:58
FaureFaure
12
edited Nov 15 '18 at 11:09
Faure
asked Nov 15 '18 at 9:58
FaureFaure
12
asked Nov 15 '18 at 9:58
FaureFaure
12
asked Nov 15 '18 at 9:58
FaureFaure
12
12
in the title you mentioned RDD (resilient distributed dataset) and later you are mentioning RRD (round robin database).. I know you probably meant RDD because of spark usage, but please be consistent..
– Dejan Marić
Nov 15 '18 at 10:22
add a comment |
in the title you mentioned RDD (resilient distributed dataset) and later you are mentioning RRD (round robin database).. I know you probably meant RDD because of spark usage, but please be consistent..
– Dejan Marić
Nov 15 '18 at 10:22
in the title you mentioned RDD (resilient distributed dataset) and later you are mentioning RRD (round robin database).. I know you probably meant RDD because of spark usage, but please be consistent..
– Dejan Marić
Nov 15 '18 at 10:22
in the title you mentioned RDD (resilient distributed dataset) and later you are mentioning RRD (round robin database).. I know you probably meant RDD because of spark usage, but please be consistent..
– Dejan Marić
Nov 15 '18 at 10:22
add a comment |
1 Answer
1
active
oldest
votes
pp_df = pp_df.rdd.map(lambda x: LabeledPoint(x[10], x[:10])).collect()returns an array not an RDD. Thus you can't use the randomSplit method on this.
Spark Actions : https://spark.apache.org/docs/latest/rdd-programming-guide.html
answered Nov 15 '18 at 10:50
rbcvlrbcvl
737
Thanks for pointing this out. Do you know how do I return a RRD labeled dataset ?
– Faure
Nov 15 '18 at 11:10
You should try something out without collect().
– rbcvl
Nov 15 '18 at 14:47
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53316770%2fhow-can-i-convert-csv-to-rdd-of-labeled-points%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
pp_df = pp_df.rdd.map(lambda x: LabeledPoint(x[10], x[:10])).collect()returns an array not an RDD. Thus you can't use the randomSplit method on this.
Spark Actions : https://spark.apache.org/docs/latest/rdd-programming-guide.html
answered Nov 15 '18 at 10:50
rbcvlrbcvl
737
Thanks for pointing this out. Do you know how do I return a RRD labeled dataset ?
– Faure
Nov 15 '18 at 11:10
You should try something out without collect().
– rbcvl
Nov 15 '18 at 14:47
add a comment |
pp_df = pp_df.rdd.map(lambda x: LabeledPoint(x[10], x[:10])).collect()returns an array not an RDD. Thus you can't use the randomSplit method on this.
Spark Actions : https://spark.apache.org/docs/latest/rdd-programming-guide.html
answered Nov 15 '18 at 10:50
rbcvlrbcvl
737
Thanks for pointing this out. Do you know how do I return a RRD labeled dataset ?
– Faure
Nov 15 '18 at 11:10
You should try something out without collect().
– rbcvl
Nov 15 '18 at 14:47
add a comment |
pp_df = pp_df.rdd.map(lambda x: LabeledPoint(x[10], x[:10])).collect()returns an array not an RDD. Thus you can't use the randomSplit method on this.
Spark Actions : https://spark.apache.org/docs/latest/rdd-programming-guide.html
answered Nov 15 '18 at 10:50
rbcvlrbcvl
737
pp_df = pp_df.rdd.map(lambda x: LabeledPoint(x[10], x[:10])).collect()returns an array not an RDD. Thus you can't use the randomSplit method on this.
Spark Actions : https://spark.apache.org/docs/latest/rdd-programming-guide.html
answered Nov 15 '18 at 10:50
rbcvlrbcvl
737
edited Nov 15 '18 at 10:56
answered Nov 15 '18 at 10:50
rbcvlrbcvl
737
answered Nov 15 '18 at 10:50
rbcvlrbcvl
737
answered Nov 15 '18 at 10:50
rbcvlrbcvl
737
737
Thanks for pointing this out. Do you know how do I return a RRD labeled dataset ?
– Faure
Nov 15 '18 at 11:10
You should try something out without collect().
– rbcvl
Nov 15 '18 at 14:47
add a comment |
Thanks for pointing this out. Do you know how do I return a RRD labeled dataset ?
– Faure
Nov 15 '18 at 11:10
You should try something out without collect().
– rbcvl
Nov 15 '18 at 14:47
Thanks for pointing this out. Do you know how do I return a RRD labeled dataset ?
– Faure
Nov 15 '18 at 11:10
Thanks for pointing this out. Do you know how do I return a RRD labeled dataset ?
– Faure
Nov 15 '18 at 11:10
You should try something out without collect().
– rbcvl
Nov 15 '18 at 14:47
You should try something out without collect().
– rbcvl
Nov 15 '18 at 14:47
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Some of your past answers have not been well-received, and you're in danger of being blocked from answering.
Please pay close attention to the following guidance:
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53316770%2fhow-can-i-convert-csv-to-rdd-of-labeled-points%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
in the title you mentioned RDD (resilient distributed dataset) and later you are mentioning RRD (round robin database).. I know you probably meant RDD because of spark usage, but please be consistent..
– Dejan Marić
Nov 15 '18 at 10:22