How does reduceByKey and mapValues works simultaneously?
I am completely new to spark and in the world of big data. I have a code which actually creates a function which splits a CSV file and returns two fields.
Then there is map function which I know how it works, but I am confused in next part of the code (operation happening on totalsByAge variable) , mapValues and reduceByKey are applying. Please help me to understand how reduceByKey and mapValues works here?
def parseLine(line):
fields = line.split(',')
age = int(fields[2])
numFriends = int(fields[3])
return (age,numFriends)
line = sparkCont.textFile("D:\ResearchInMotion\ml-100k\fakefriends.csv")
rdd = line.map(parseLine)
totalsByAge = rdd.mapValues(lambda x: (x, 1)).reduceByKey(lambda x, y: (x[0] + y[0], x[1] + y[1]))
averagesByAge = totalsByAge.mapValues(lambda x: x[0] / x[1])
results = averagesByAge.collect()
for result in results:
print(result)
I need help in totalsByAge variable processing.It would be good if you can also elaborate the operation done on averagesByAge Please let me know if anything is missing.
python apache-spark pyspark rdd
asked Nov 21 '18 at 13:32
Sahil NagpalSahil Nagpal
351311
add a comment |
I am completely new to spark and in the world of big data. I have a code which actually creates a function which splits a CSV file and returns two fields.
Then there is map function which I know how it works, but I am confused in next part of the code (operation happening on totalsByAge variable) , mapValues and reduceByKey are applying. Please help me to understand how reduceByKey and mapValues works here?
def parseLine(line):
fields = line.split(',')
age = int(fields[2])
numFriends = int(fields[3])
return (age,numFriends)
line = sparkCont.textFile("D:\ResearchInMotion\ml-100k\fakefriends.csv")
rdd = line.map(parseLine)
totalsByAge = rdd.mapValues(lambda x: (x, 1)).reduceByKey(lambda x, y: (x[0] + y[0], x[1] + y[1]))
averagesByAge = totalsByAge.mapValues(lambda x: x[0] / x[1])
results = averagesByAge.collect()
for result in results:
print(result)
I need help in totalsByAge variable processing.It would be good if you can also elaborate the operation done on averagesByAge Please let me know if anything is missing.
python apache-spark pyspark rdd
asked Nov 21 '18 at 13:32
Sahil NagpalSahil Nagpal
351311
add a comment |
I am completely new to spark and in the world of big data. I have a code which actually creates a function which splits a CSV file and returns two fields.
Then there is map function which I know how it works, but I am confused in next part of the code (operation happening on totalsByAge variable) , mapValues and reduceByKey are applying. Please help me to understand how reduceByKey and mapValues works here?
def parseLine(line):
fields = line.split(',')
age = int(fields[2])
numFriends = int(fields[3])
return (age,numFriends)
line = sparkCont.textFile("D:\ResearchInMotion\ml-100k\fakefriends.csv")
rdd = line.map(parseLine)
totalsByAge = rdd.mapValues(lambda x: (x, 1)).reduceByKey(lambda x, y: (x[0] + y[0], x[1] + y[1]))
averagesByAge = totalsByAge.mapValues(lambda x: x[0] / x[1])
results = averagesByAge.collect()
for result in results:
print(result)
I need help in totalsByAge variable processing.It would be good if you can also elaborate the operation done on averagesByAge Please let me know if anything is missing.
python apache-spark pyspark rdd
asked Nov 21 '18 at 13:32
Sahil NagpalSahil Nagpal
351311
I am completely new to spark and in the world of big data. I have a code which actually creates a function which splits a CSV file and returns two fields.
Then there is map function which I know how it works, but I am confused in next part of the code (operation happening on totalsByAge variable) , mapValues and reduceByKey are applying. Please help me to understand how reduceByKey and mapValues works here?
def parseLine(line):
fields = line.split(',')
age = int(fields[2])
numFriends = int(fields[3])
return (age,numFriends)
line = sparkCont.textFile("D:\ResearchInMotion\ml-100k\fakefriends.csv")
rdd = line.map(parseLine)
totalsByAge = rdd.mapValues(lambda x: (x, 1)).reduceByKey(lambda x, y: (x[0] + y[0], x[1] + y[1]))
averagesByAge = totalsByAge.mapValues(lambda x: x[0] / x[1])
results = averagesByAge.collect()
for result in results:
print(result)
I need help in totalsByAge variable processing.It would be good if you can also elaborate the operation done on averagesByAge Please let me know if anything is missing.
python apache-spark pyspark rdd
python apache-spark pyspark rdd
asked Nov 21 '18 at 13:32
Sahil NagpalSahil Nagpal
351311
asked Nov 21 '18 at 13:32
Sahil NagpalSahil Nagpal
351311
asked Nov 21 '18 at 13:32
Sahil NagpalSahil Nagpal
351311
asked Nov 21 '18 at 13:32
Sahil NagpalSahil Nagpal
351311
asked Nov 21 '18 at 13:32
Sahil NagpalSahil Nagpal
351311
351311
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
In the line of rdd = line.map(parseLine) you have pair of values in the format of (age, numFriends) like (a_1, n_1), (a_2, n_2), ..., (a_m, n_m). In rdd.mapValues(lambda x: (x, 1)) you will get (a_1, (n_1, 1)), (a_2, (n_2, 1)), ..., (a_m, (n_m, 1)).
In reduceByKey, first grouped by key, it means all the same age grouped in a group and you will have something likes (a_i, iterator over pairs of (n_j, 1) which all n_j has the same age), and after that apply the function of the reduction. And reduction part means sum all numFriends with each other for each age, and 1s with each other, which sum of 1s means the number items in the list.
Therefore, after reduceByKey, we will have (a_i, (sum of all numFriends in the list, number of items in the list)). In the otherwords, the first value of the outside pair is age and second value is a inside pair which its first value is sum of all numFriends and the second value is the number of items. Hence, totalsByAge.mapValues(lambda x: x[0] / x[1]) gives us average of numFriends for each age.
answered Nov 21 '18 at 15:01
OmGOmG
8,46953047
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53413210%2fhow-does-reducebykey-and-mapvalues-works-simultaneously%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
In the line of rdd = line.map(parseLine) you have pair of values in the format of (age, numFriends) like (a_1, n_1), (a_2, n_2), ..., (a_m, n_m). In rdd.mapValues(lambda x: (x, 1)) you will get (a_1, (n_1, 1)), (a_2, (n_2, 1)), ..., (a_m, (n_m, 1)).
In reduceByKey, first grouped by key, it means all the same age grouped in a group and you will have something likes (a_i, iterator over pairs of (n_j, 1) which all n_j has the same age), and after that apply the function of the reduction. And reduction part means sum all numFriends with each other for each age, and 1s with each other, which sum of 1s means the number items in the list.
Therefore, after reduceByKey, we will have (a_i, (sum of all numFriends in the list, number of items in the list)). In the otherwords, the first value of the outside pair is age and second value is a inside pair which its first value is sum of all numFriends and the second value is the number of items. Hence, totalsByAge.mapValues(lambda x: x[0] / x[1]) gives us average of numFriends for each age.
answered Nov 21 '18 at 15:01
OmGOmG
8,46953047
add a comment |
In the line of rdd = line.map(parseLine) you have pair of values in the format of (age, numFriends) like (a_1, n_1), (a_2, n_2), ..., (a_m, n_m). In rdd.mapValues(lambda x: (x, 1)) you will get (a_1, (n_1, 1)), (a_2, (n_2, 1)), ..., (a_m, (n_m, 1)).
In reduceByKey, first grouped by key, it means all the same age grouped in a group and you will have something likes (a_i, iterator over pairs of (n_j, 1) which all n_j has the same age), and after that apply the function of the reduction. And reduction part means sum all numFriends with each other for each age, and 1s with each other, which sum of 1s means the number items in the list.
Therefore, after reduceByKey, we will have (a_i, (sum of all numFriends in the list, number of items in the list)). In the otherwords, the first value of the outside pair is age and second value is a inside pair which its first value is sum of all numFriends and the second value is the number of items. Hence, totalsByAge.mapValues(lambda x: x[0] / x[1]) gives us average of numFriends for each age.
answered Nov 21 '18 at 15:01
OmGOmG
8,46953047
add a comment |
In the line of rdd = line.map(parseLine) you have pair of values in the format of (age, numFriends) like (a_1, n_1), (a_2, n_2), ..., (a_m, n_m). In rdd.mapValues(lambda x: (x, 1)) you will get (a_1, (n_1, 1)), (a_2, (n_2, 1)), ..., (a_m, (n_m, 1)).
In reduceByKey, first grouped by key, it means all the same age grouped in a group and you will have something likes (a_i, iterator over pairs of (n_j, 1) which all n_j has the same age), and after that apply the function of the reduction. And reduction part means sum all numFriends with each other for each age, and 1s with each other, which sum of 1s means the number items in the list.
Therefore, after reduceByKey, we will have (a_i, (sum of all numFriends in the list, number of items in the list)). In the otherwords, the first value of the outside pair is age and second value is a inside pair which its first value is sum of all numFriends and the second value is the number of items. Hence, totalsByAge.mapValues(lambda x: x[0] / x[1]) gives us average of numFriends for each age.
answered Nov 21 '18 at 15:01
OmGOmG
8,46953047
In the line of rdd = line.map(parseLine) you have pair of values in the format of (age, numFriends) like (a_1, n_1), (a_2, n_2), ..., (a_m, n_m). In rdd.mapValues(lambda x: (x, 1)) you will get (a_1, (n_1, 1)), (a_2, (n_2, 1)), ..., (a_m, (n_m, 1)).
In reduceByKey, first grouped by key, it means all the same age grouped in a group and you will have something likes (a_i, iterator over pairs of (n_j, 1) which all n_j has the same age), and after that apply the function of the reduction. And reduction part means sum all numFriends with each other for each age, and 1s with each other, which sum of 1s means the number items in the list.
Therefore, after reduceByKey, we will have (a_i, (sum of all numFriends in the list, number of items in the list)). In the otherwords, the first value of the outside pair is age and second value is a inside pair which its first value is sum of all numFriends and the second value is the number of items. Hence, totalsByAge.mapValues(lambda x: x[0] / x[1]) gives us average of numFriends for each age.
answered Nov 21 '18 at 15:01
OmGOmG
8,46953047
answered Nov 21 '18 at 15:01
OmGOmG
8,46953047
answered Nov 21 '18 at 15:01
OmGOmG
8,46953047
answered Nov 21 '18 at 15:01
OmGOmG
8,46953047
8,46953047
add a comment |
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53413210%2fhow-does-reducebykey-and-mapvalues-works-simultaneously%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
