How does the pandas read_csv parse regular expressions, exactly?
I have a CSV file with the following structure:
word1|word2|word3,word4,0.20,0.20,0.11,0.54,2.70,0.07,1.75
That is, a first column of strings (some capitalized, some not) that are separated by '|' and by ',' (this marks differences in patterns of association) and then 7 digits each separated by ','.
n.b. This dataframe has multiple millions of rows. I have tried to load it as follows:
pd.read_csv('pattern_association.csv',sep= ',(?!D)', engine='python',chunksize=10000)
I have followed the advice on here to use a regular expression which aims to capture every column after a digit, but I need one that both selects the first column as a whole string and ignores commas between strings, and then also parses out the 7 columns that are comprised of digits.
How can I get pandas to parse this?
I always get the error.
Error could possibly be due to quotes being ignored when a
multi-char delimiter is used.
I have tried many variations and the regex I am using seems to work outside the context of pandas on toy expressions.
Thanks for any tips.
python regex pandas
edited Nov 20 '18 at 2:05
Aqueous Carlos
364314
asked Nov 20 '18 at 1:48
capt_ahabcapt_ahab
61
add a comment |
I have a CSV file with the following structure:
word1|word2|word3,word4,0.20,0.20,0.11,0.54,2.70,0.07,1.75
That is, a first column of strings (some capitalized, some not) that are separated by '|' and by ',' (this marks differences in patterns of association) and then 7 digits each separated by ','.
n.b. This dataframe has multiple millions of rows. I have tried to load it as follows:
pd.read_csv('pattern_association.csv',sep= ',(?!D)', engine='python',chunksize=10000)
I have followed the advice on here to use a regular expression which aims to capture every column after a digit, but I need one that both selects the first column as a whole string and ignores commas between strings, and then also parses out the 7 columns that are comprised of digits.
How can I get pandas to parse this?
I always get the error.
Error could possibly be due to quotes being ignored when a
multi-char delimiter is used.
I have tried many variations and the regex I am using seems to work outside the context of pandas on toy expressions.
Thanks for any tips.
python regex pandas
edited Nov 20 '18 at 2:05
Aqueous Carlos
364314
asked Nov 20 '18 at 1:48
capt_ahabcapt_ahab
61
The distinction (between | and ,) contains meaningful information.
– capt_ahab
Nov 20 '18 at 2:06
1
It is not clear what the desired output is. Can you add a few lines of sample input with the kind of variation you are expecting and include the desired dataframe you want to get from that input? For example, it is not clear if the number of pipes and commas is always the same.
– Paulo Scardine
Nov 20 '18 at 2:28
add a comment |
I have a CSV file with the following structure:
word1|word2|word3,word4,0.20,0.20,0.11,0.54,2.70,0.07,1.75
That is, a first column of strings (some capitalized, some not) that are separated by '|' and by ',' (this marks differences in patterns of association) and then 7 digits each separated by ','.
n.b. This dataframe has multiple millions of rows. I have tried to load it as follows:
pd.read_csv('pattern_association.csv',sep= ',(?!D)', engine='python',chunksize=10000)
I have followed the advice on here to use a regular expression which aims to capture every column after a digit, but I need one that both selects the first column as a whole string and ignores commas between strings, and then also parses out the 7 columns that are comprised of digits.
How can I get pandas to parse this?
I always get the error.
Error could possibly be due to quotes being ignored when a
multi-char delimiter is used.
I have tried many variations and the regex I am using seems to work outside the context of pandas on toy expressions.
Thanks for any tips.
python regex pandas
edited Nov 20 '18 at 2:05
Aqueous Carlos
364314
asked Nov 20 '18 at 1:48
capt_ahabcapt_ahab
61
I have a CSV file with the following structure:
word1|word2|word3,word4,0.20,0.20,0.11,0.54,2.70,0.07,1.75
That is, a first column of strings (some capitalized, some not) that are separated by '|' and by ',' (this marks differences in patterns of association) and then 7 digits each separated by ','.
n.b. This dataframe has multiple millions of rows. I have tried to load it as follows:
pd.read_csv('pattern_association.csv',sep= ',(?!D)', engine='python',chunksize=10000)
I have followed the advice on here to use a regular expression which aims to capture every column after a digit, but I need one that both selects the first column as a whole string and ignores commas between strings, and then also parses out the 7 columns that are comprised of digits.
How can I get pandas to parse this?
I always get the error.
Error could possibly be due to quotes being ignored when a
multi-char delimiter is used.
I have tried many variations and the regex I am using seems to work outside the context of pandas on toy expressions.
Thanks for any tips.
python regex pandas
python regex pandas
edited Nov 20 '18 at 2:05
Aqueous Carlos
364314
asked Nov 20 '18 at 1:48
capt_ahabcapt_ahab
61
edited Nov 20 '18 at 2:05
Aqueous Carlos
364314
asked Nov 20 '18 at 1:48
capt_ahabcapt_ahab
61
edited Nov 20 '18 at 2:05
Aqueous Carlos
364314
edited Nov 20 '18 at 2:05
Aqueous Carlos
364314
edited Nov 20 '18 at 2:05
Aqueous Carlos
364314
364314
asked Nov 20 '18 at 1:48
capt_ahabcapt_ahab
61
asked Nov 20 '18 at 1:48
capt_ahabcapt_ahab
61
asked Nov 20 '18 at 1:48
capt_ahabcapt_ahab
61
61
The distinction (between | and ,) contains meaningful information.
– capt_ahab
Nov 20 '18 at 2:06
1
It is not clear what the desired output is. Can you add a few lines of sample input with the kind of variation you are expecting and include the desired dataframe you want to get from that input? For example, it is not clear if the number of pipes and commas is always the same.
– Paulo Scardine
Nov 20 '18 at 2:28
add a comment |
The distinction (between | and ,) contains meaningful information.
– capt_ahab
Nov 20 '18 at 2:06
1
It is not clear what the desired output is. Can you add a few lines of sample input with the kind of variation you are expecting and include the desired dataframe you want to get from that input? For example, it is not clear if the number of pipes and commas is always the same.
– Paulo Scardine
Nov 20 '18 at 2:28
The distinction (between | and ,) contains meaningful information.
– capt_ahab
Nov 20 '18 at 2:06
The distinction (between | and ,) contains meaningful information.
– capt_ahab
Nov 20 '18 at 2:06
1
1
It is not clear what the desired output is. Can you add a few lines of sample input with the kind of variation you are expecting and include the desired dataframe you want to get from that input? For example, it is not clear if the number of pipes and commas is always the same.
– Paulo Scardine
Nov 20 '18 at 2:28
It is not clear what the desired output is. Can you add a few lines of sample input with the kind of variation you are expecting and include the desired dataframe you want to get from that input? For example, it is not clear if the number of pipes and commas is always the same.
– Paulo Scardine
Nov 20 '18 at 2:28
add a comment |
0
active
oldest
votes
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53385104%2fhow-does-the-pandas-read-csv-parse-regular-expressions-exactly%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
0
active
oldest
votes
0
active
oldest
votes
active
oldest
votes
active
oldest
votes
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53385104%2fhow-does-the-pandas-read-csv-parse-regular-expressions-exactly%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
The distinction (between | and ,) contains meaningful information.
– capt_ahab
Nov 20 '18 at 2:06
1
It is not clear what the desired output is. Can you add a few lines of sample input with the kind of variation you are expecting and include the desired dataframe you want to get from that input? For example, it is not clear if the number of pipes and commas is always the same.
– Paulo Scardine
Nov 20 '18 at 2:28