df.fillna() not working but df.dropna() working
.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty{ height:90px;width:728px;box-sizing:border-box;
}
0
In my code the df.fillna() method is not working when the df.dropna() method is working. I don't want to drop the column though. What can I do that the fillna() method works?
def preprocess_df(df):
for col in df.columns: # go through all of the columns
if col != "target": # normalize all ... except for the target itself!
df[col] = df[col].pct_change() # pct change "normalizes" the different currencies (each crypto coin has vastly diff values, we're really more interested in the other coin's movements)
# df.dropna(inplace=True) # remove the nas created by pct_change
df.fillna(method="ffill", inplace=True)
print(df)
break
df[col] = preprocessing.scale(df[col].values) # scale between 0 and 1.
python pandas
asked Nov 22 '18 at 14:11
user9468014user9468014
5110
|
show 1 more comment
0
In my code the df.fillna() method is not working when the df.dropna() method is working. I don't want to drop the column though. What can I do that the fillna() method works?
def preprocess_df(df):
for col in df.columns: # go through all of the columns
if col != "target": # normalize all ... except for the target itself!
df[col] = df[col].pct_change() # pct change "normalizes" the different currencies (each crypto coin has vastly diff values, we're really more interested in the other coin's movements)
# df.dropna(inplace=True) # remove the nas created by pct_change
df.fillna(method="ffill", inplace=True)
print(df)
break
df[col] = preprocessing.scale(df[col].values) # scale between 0 and 1.
python pandas
asked Nov 22 '18 at 14:11
user9468014user9468014
5110
Trydf.fillna(method="ffill", inplace=True, axis=1)
– roganjosh
Nov 22 '18 at 14:16
3
What do you mean by "not working"? Are you getting an error? If so, what's the error message?
– Lukas Thaler
Nov 22 '18 at 14:16
the NaN does not disappear.
– user9468014
Nov 22 '18 at 14:18
I've tried with df.fillna(method="ffill", inplace=True, axis=1) but still the same.
– user9468014
Nov 22 '18 at 14:18
Error message is: ValueError: Input contains NaN, infinity or a value too large for dtype('float64').
– user9468014
Nov 22 '18 at 14:20
|
show 1 more comment
0
0
0
0
In my code the df.fillna() method is not working when the df.dropna() method is working. I don't want to drop the column though. What can I do that the fillna() method works?
def preprocess_df(df):
for col in df.columns: # go through all of the columns
if col != "target": # normalize all ... except for the target itself!
df[col] = df[col].pct_change() # pct change "normalizes" the different currencies (each crypto coin has vastly diff values, we're really more interested in the other coin's movements)
# df.dropna(inplace=True) # remove the nas created by pct_change
df.fillna(method="ffill", inplace=True)
print(df)
break
df[col] = preprocessing.scale(df[col].values) # scale between 0 and 1.
python pandas
asked Nov 22 '18 at 14:11
user9468014user9468014
5110
In my code the df.fillna() method is not working when the df.dropna() method is working. I don't want to drop the column though. What can I do that the fillna() method works?
def preprocess_df(df):
for col in df.columns: # go through all of the columns
if col != "target": # normalize all ... except for the target itself!
df[col] = df[col].pct_change() # pct change "normalizes" the different currencies (each crypto coin has vastly diff values, we're really more interested in the other coin's movements)
# df.dropna(inplace=True) # remove the nas created by pct_change
df.fillna(method="ffill", inplace=True)
print(df)
break
df[col] = preprocessing.scale(df[col].values) # scale between 0 and 1.
python pandas
python pandas
asked Nov 22 '18 at 14:11
user9468014user9468014
5110
asked Nov 22 '18 at 14:11
user9468014user9468014
5110
asked Nov 22 '18 at 14:11
user9468014user9468014
5110
asked Nov 22 '18 at 14:11
user9468014user9468014
5110
asked Nov 22 '18 at 14:11
user9468014user9468014
5110
5110
Trydf.fillna(method="ffill", inplace=True, axis=1)
– roganjosh
Nov 22 '18 at 14:16
3
What do you mean by "not working"? Are you getting an error? If so, what's the error message?
– Lukas Thaler
Nov 22 '18 at 14:16
the NaN does not disappear.
– user9468014
Nov 22 '18 at 14:18
I've tried with df.fillna(method="ffill", inplace=True, axis=1) but still the same.
– user9468014
Nov 22 '18 at 14:18
Error message is: ValueError: Input contains NaN, infinity or a value too large for dtype('float64').
– user9468014
Nov 22 '18 at 14:20
|
show 1 more comment
Trydf.fillna(method="ffill", inplace=True, axis=1)
– roganjosh
Nov 22 '18 at 14:16
3
What do you mean by "not working"? Are you getting an error? If so, what's the error message?
– Lukas Thaler
Nov 22 '18 at 14:16
the NaN does not disappear.
– user9468014
Nov 22 '18 at 14:18
I've tried with df.fillna(method="ffill", inplace=True, axis=1) but still the same.
– user9468014
Nov 22 '18 at 14:18
Error message is: ValueError: Input contains NaN, infinity or a value too large for dtype('float64').
– user9468014
Nov 22 '18 at 14:20
Try
df.fillna(method="ffill", inplace=True, axis=1)– roganjosh
Nov 22 '18 at 14:16
Try
df.fillna(method="ffill", inplace=True, axis=1)– roganjosh
Nov 22 '18 at 14:16
3
3
What do you mean by "not working"? Are you getting an error? If so, what's the error message?
– Lukas Thaler
Nov 22 '18 at 14:16
What do you mean by "not working"? Are you getting an error? If so, what's the error message?
– Lukas Thaler
Nov 22 '18 at 14:16
the NaN does not disappear.
– user9468014
Nov 22 '18 at 14:18
the NaN does not disappear.
– user9468014
Nov 22 '18 at 14:18
I've tried with df.fillna(method="ffill", inplace=True, axis=1) but still the same.
– user9468014
Nov 22 '18 at 14:18
I've tried with df.fillna(method="ffill", inplace=True, axis=1) but still the same.
– user9468014
Nov 22 '18 at 14:18
Error message is: ValueError: Input contains NaN, infinity or a value too large for dtype('float64').
– user9468014
Nov 22 '18 at 14:20
Error message is: ValueError: Input contains NaN, infinity or a value too large for dtype('float64').
– user9468014
Nov 22 '18 at 14:20
|
show 1 more comment
1 Answer
1
active
oldest
votes
0
You were almost there:
df = df.fillna(method="ffill", inplace=True)
You have to assign it back to df
answered Nov 22 '18 at 15:01
erncyperncyp
336311
3
This is incorrect. You should not assign back when you useinplace=True.
– jpp
Nov 22 '18 at 15:09
not working unfortunetally :(
– user9468014
Nov 22 '18 at 22:45
My bad @jpp, you are correct. Maybe it is something to do withmethod='ffill'? According to pd docs,ffillpropagates the last valid observation. What if the entire column is NaN s. How does it know what to propagate it with? Perhaps its worth manually telling it what to fill the NaNs with?
– erncyp
Nov 23 '18 at 9:15
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
draft saved
draft discarded
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53432838%2fdf-fillna-not-working-but-df-dropna-working%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
0
You were almost there:
df = df.fillna(method="ffill", inplace=True)
You have to assign it back to df
answered Nov 22 '18 at 15:01
erncyperncyp
336311
3
This is incorrect. You should not assign back when you useinplace=True.
– jpp
Nov 22 '18 at 15:09
not working unfortunetally :(
– user9468014
Nov 22 '18 at 22:45
My bad @jpp, you are correct. Maybe it is something to do withmethod='ffill'? According to pd docs,ffillpropagates the last valid observation. What if the entire column is NaN s. How does it know what to propagate it with? Perhaps its worth manually telling it what to fill the NaNs with?
– erncyp
Nov 23 '18 at 9:15
add a comment |
0
You were almost there:
df = df.fillna(method="ffill", inplace=True)
You have to assign it back to df
answered Nov 22 '18 at 15:01
erncyperncyp
336311
3
This is incorrect. You should not assign back when you useinplace=True.
– jpp
Nov 22 '18 at 15:09
not working unfortunetally :(
– user9468014
Nov 22 '18 at 22:45
My bad @jpp, you are correct. Maybe it is something to do withmethod='ffill'? According to pd docs,ffillpropagates the last valid observation. What if the entire column is NaN s. How does it know what to propagate it with? Perhaps its worth manually telling it what to fill the NaNs with?
– erncyp
Nov 23 '18 at 9:15
add a comment |
0
0
0
You were almost there:
df = df.fillna(method="ffill", inplace=True)
You have to assign it back to df
answered Nov 22 '18 at 15:01
erncyperncyp
336311
You were almost there:
df = df.fillna(method="ffill", inplace=True)
You have to assign it back to df
answered Nov 22 '18 at 15:01
erncyperncyp
336311
answered Nov 22 '18 at 15:01
erncyperncyp
336311
answered Nov 22 '18 at 15:01
erncyperncyp
336311
answered Nov 22 '18 at 15:01
erncyperncyp
336311
336311
3
This is incorrect. You should not assign back when you useinplace=True.
– jpp
Nov 22 '18 at 15:09
not working unfortunetally :(
– user9468014
Nov 22 '18 at 22:45
My bad @jpp, you are correct. Maybe it is something to do withmethod='ffill'? According to pd docs,ffillpropagates the last valid observation. What if the entire column is NaN s. How does it know what to propagate it with? Perhaps its worth manually telling it what to fill the NaNs with?
– erncyp
Nov 23 '18 at 9:15
add a comment |
3
This is incorrect. You should not assign back when you useinplace=True.
– jpp
Nov 22 '18 at 15:09
not working unfortunetally :(
– user9468014
Nov 22 '18 at 22:45
My bad @jpp, you are correct. Maybe it is something to do withmethod='ffill'? According to pd docs,ffillpropagates the last valid observation. What if the entire column is NaN s. How does it know what to propagate it with? Perhaps its worth manually telling it what to fill the NaNs with?
– erncyp
Nov 23 '18 at 9:15
3
3
This is incorrect. You should not assign back when you use
inplace=True.– jpp
Nov 22 '18 at 15:09
This is incorrect. You should not assign back when you use
inplace=True.– jpp
Nov 22 '18 at 15:09
not working unfortunetally :(
– user9468014
Nov 22 '18 at 22:45
not working unfortunetally :(
– user9468014
Nov 22 '18 at 22:45
My bad @jpp, you are correct. Maybe it is something to do with
method='ffill'? According to pd docs, ffill propagates the last valid observation. What if the entire column is NaN s. How does it know what to propagate it with? Perhaps its worth manually telling it what to fill the NaNs with?– erncyp
Nov 23 '18 at 9:15
My bad @jpp, you are correct. Maybe it is something to do with
method='ffill'? According to pd docs, ffill propagates the last valid observation. What if the entire column is NaN s. How does it know what to propagate it with? Perhaps its worth manually telling it what to fill the NaNs with?– erncyp
Nov 23 '18 at 9:15
add a comment |
draft saved
draft discarded
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
draft saved
draft discarded
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53432838%2fdf-fillna-not-working-but-df-dropna-working%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Try
df.fillna(method="ffill", inplace=True, axis=1)– roganjosh
Nov 22 '18 at 14:16
3
What do you mean by "not working"? Are you getting an error? If so, what's the error message?
– Lukas Thaler
Nov 22 '18 at 14:16
the NaN does not disappear.
– user9468014
Nov 22 '18 at 14:18
I've tried with df.fillna(method="ffill", inplace=True, axis=1) but still the same.
– user9468014
Nov 22 '18 at 14:18
Error message is: ValueError: Input contains NaN, infinity or a value too large for dtype('float64').
– user9468014
Nov 22 '18 at 14:20