Paramiko Download, process and re-upload the same file











up vote
1
down vote

favorite
1












I am using Paramiko to create an SFTP client to create a backup copy of a JSON file, read in the contents of the original, then update (the original). I am able to get this snippet of code to work:



# open sftp connection stuff

# read in json create backup copy - but have to 'open' twice
read_file = sftp_client.open(file_path)
settings = json.load(read_file)
read_file = sftp_client.open(file_path)
sftp_client.putfo(read_file, backup_path)

# json stuff and updating
new_settings = json.dumps(settings, indent=4, sort_keys = True)

# update remote json file
with sftp_client.open(file_path, 'w') as f:
f.write(new_settings)


However when I try to clean up the code and combine the backup file creation and JSON load:



with sftp_client.open(file_path) as f:
sftp_client.putfo(f, backup_path)
settings = json.load(f)


The backup file will be created but json.load will fail to due not having any content. And if I reverse the order, the json.load will read in the values, but the backup copy will be empty.



I'm using Python 2.7 on a Windows machine, creating a remote connection to a QNX (Linux) machine. Appreciate any help.



Thanks in advance.










share|improve this question
























  • "I am avoiding the 'put' command to preserve the ascii/text format of the original file." - That makes no sense.
    – Martin Prikryl
    Nov 11 at 20:00










  • Retrieving and the 'putting' the file back as a binary puts a lot of ctrl-m characters at the end of lines. I'd prefer not to have to search and replace in VI to remove them. Although I am not sure if ^M will mess up the processes that read in the json file in question. Just wanted to be safe...
    – canofwhoopass
    Nov 11 at 20:35










  • Also want to add that my limited understanding is that paramiko only 'puts' in binary mode. If there is a way to get it in ascii mode, that would simplify things.
    – canofwhoopass
    Nov 11 at 20:43










  • Quite opposite, retrieving a file and putting it back in the binary mode cannot change the file in any way. That's the principle of the binary mode. Moreover, there's no difference between using put and putfo in this respect. Actually all that put does is that it calls open and putfo, just like your code does.
    – Martin Prikryl
    Nov 11 at 22:39












  • Thanks for the tip - complete novice here at this end. Didn't know how to explain why my new file had ^M markers when the originals didn't if the real SW guys asked.
    – canofwhoopass
    Nov 12 at 2:54

















up vote
1
down vote

favorite
1












I am using Paramiko to create an SFTP client to create a backup copy of a JSON file, read in the contents of the original, then update (the original). I am able to get this snippet of code to work:



# open sftp connection stuff

# read in json create backup copy - but have to 'open' twice
read_file = sftp_client.open(file_path)
settings = json.load(read_file)
read_file = sftp_client.open(file_path)
sftp_client.putfo(read_file, backup_path)

# json stuff and updating
new_settings = json.dumps(settings, indent=4, sort_keys = True)

# update remote json file
with sftp_client.open(file_path, 'w') as f:
f.write(new_settings)


However when I try to clean up the code and combine the backup file creation and JSON load:



with sftp_client.open(file_path) as f:
sftp_client.putfo(f, backup_path)
settings = json.load(f)


The backup file will be created but json.load will fail to due not having any content. And if I reverse the order, the json.load will read in the values, but the backup copy will be empty.



I'm using Python 2.7 on a Windows machine, creating a remote connection to a QNX (Linux) machine. Appreciate any help.



Thanks in advance.










share|improve this question
























  • "I am avoiding the 'put' command to preserve the ascii/text format of the original file." - That makes no sense.
    – Martin Prikryl
    Nov 11 at 20:00










  • Retrieving and the 'putting' the file back as a binary puts a lot of ctrl-m characters at the end of lines. I'd prefer not to have to search and replace in VI to remove them. Although I am not sure if ^M will mess up the processes that read in the json file in question. Just wanted to be safe...
    – canofwhoopass
    Nov 11 at 20:35










  • Also want to add that my limited understanding is that paramiko only 'puts' in binary mode. If there is a way to get it in ascii mode, that would simplify things.
    – canofwhoopass
    Nov 11 at 20:43










  • Quite opposite, retrieving a file and putting it back in the binary mode cannot change the file in any way. That's the principle of the binary mode. Moreover, there's no difference between using put and putfo in this respect. Actually all that put does is that it calls open and putfo, just like your code does.
    – Martin Prikryl
    Nov 11 at 22:39












  • Thanks for the tip - complete novice here at this end. Didn't know how to explain why my new file had ^M markers when the originals didn't if the real SW guys asked.
    – canofwhoopass
    Nov 12 at 2:54















up vote
1
down vote

favorite
1









up vote
1
down vote

favorite
1






1





I am using Paramiko to create an SFTP client to create a backup copy of a JSON file, read in the contents of the original, then update (the original). I am able to get this snippet of code to work:



# open sftp connection stuff

# read in json create backup copy - but have to 'open' twice
read_file = sftp_client.open(file_path)
settings = json.load(read_file)
read_file = sftp_client.open(file_path)
sftp_client.putfo(read_file, backup_path)

# json stuff and updating
new_settings = json.dumps(settings, indent=4, sort_keys = True)

# update remote json file
with sftp_client.open(file_path, 'w') as f:
f.write(new_settings)


However when I try to clean up the code and combine the backup file creation and JSON load:



with sftp_client.open(file_path) as f:
sftp_client.putfo(f, backup_path)
settings = json.load(f)


The backup file will be created but json.load will fail to due not having any content. And if I reverse the order, the json.load will read in the values, but the backup copy will be empty.



I'm using Python 2.7 on a Windows machine, creating a remote connection to a QNX (Linux) machine. Appreciate any help.



Thanks in advance.










share|improve this question















I am using Paramiko to create an SFTP client to create a backup copy of a JSON file, read in the contents of the original, then update (the original). I am able to get this snippet of code to work:



# open sftp connection stuff

# read in json create backup copy - but have to 'open' twice
read_file = sftp_client.open(file_path)
settings = json.load(read_file)
read_file = sftp_client.open(file_path)
sftp_client.putfo(read_file, backup_path)

# json stuff and updating
new_settings = json.dumps(settings, indent=4, sort_keys = True)

# update remote json file
with sftp_client.open(file_path, 'w') as f:
f.write(new_settings)


However when I try to clean up the code and combine the backup file creation and JSON load:



with sftp_client.open(file_path) as f:
sftp_client.putfo(f, backup_path)
settings = json.load(f)


The backup file will be created but json.load will fail to due not having any content. And if I reverse the order, the json.load will read in the values, but the backup copy will be empty.



I'm using Python 2.7 on a Windows machine, creating a remote connection to a QNX (Linux) machine. Appreciate any help.



Thanks in advance.







python sftp paramiko






share|improve this question















share|improve this question













share|improve this question




share|improve this question








edited Nov 12 at 7:00









Martin Prikryl

84k22159351




84k22159351










asked Nov 11 at 19:09









canofwhoopass

155




155












  • "I am avoiding the 'put' command to preserve the ascii/text format of the original file." - That makes no sense.
    – Martin Prikryl
    Nov 11 at 20:00










  • Retrieving and the 'putting' the file back as a binary puts a lot of ctrl-m characters at the end of lines. I'd prefer not to have to search and replace in VI to remove them. Although I am not sure if ^M will mess up the processes that read in the json file in question. Just wanted to be safe...
    – canofwhoopass
    Nov 11 at 20:35










  • Also want to add that my limited understanding is that paramiko only 'puts' in binary mode. If there is a way to get it in ascii mode, that would simplify things.
    – canofwhoopass
    Nov 11 at 20:43










  • Quite opposite, retrieving a file and putting it back in the binary mode cannot change the file in any way. That's the principle of the binary mode. Moreover, there's no difference between using put and putfo in this respect. Actually all that put does is that it calls open and putfo, just like your code does.
    – Martin Prikryl
    Nov 11 at 22:39












  • Thanks for the tip - complete novice here at this end. Didn't know how to explain why my new file had ^M markers when the originals didn't if the real SW guys asked.
    – canofwhoopass
    Nov 12 at 2:54




















  • "I am avoiding the 'put' command to preserve the ascii/text format of the original file." - That makes no sense.
    – Martin Prikryl
    Nov 11 at 20:00










  • Retrieving and the 'putting' the file back as a binary puts a lot of ctrl-m characters at the end of lines. I'd prefer not to have to search and replace in VI to remove them. Although I am not sure if ^M will mess up the processes that read in the json file in question. Just wanted to be safe...
    – canofwhoopass
    Nov 11 at 20:35










  • Also want to add that my limited understanding is that paramiko only 'puts' in binary mode. If there is a way to get it in ascii mode, that would simplify things.
    – canofwhoopass
    Nov 11 at 20:43










  • Quite opposite, retrieving a file and putting it back in the binary mode cannot change the file in any way. That's the principle of the binary mode. Moreover, there's no difference between using put and putfo in this respect. Actually all that put does is that it calls open and putfo, just like your code does.
    – Martin Prikryl
    Nov 11 at 22:39












  • Thanks for the tip - complete novice here at this end. Didn't know how to explain why my new file had ^M markers when the originals didn't if the real SW guys asked.
    – canofwhoopass
    Nov 12 at 2:54


















"I am avoiding the 'put' command to preserve the ascii/text format of the original file." - That makes no sense.
– Martin Prikryl
Nov 11 at 20:00




"I am avoiding the 'put' command to preserve the ascii/text format of the original file." - That makes no sense.
– Martin Prikryl
Nov 11 at 20:00












Retrieving and the 'putting' the file back as a binary puts a lot of ctrl-m characters at the end of lines. I'd prefer not to have to search and replace in VI to remove them. Although I am not sure if ^M will mess up the processes that read in the json file in question. Just wanted to be safe...
– canofwhoopass
Nov 11 at 20:35




Retrieving and the 'putting' the file back as a binary puts a lot of ctrl-m characters at the end of lines. I'd prefer not to have to search and replace in VI to remove them. Although I am not sure if ^M will mess up the processes that read in the json file in question. Just wanted to be safe...
– canofwhoopass
Nov 11 at 20:35












Also want to add that my limited understanding is that paramiko only 'puts' in binary mode. If there is a way to get it in ascii mode, that would simplify things.
– canofwhoopass
Nov 11 at 20:43




Also want to add that my limited understanding is that paramiko only 'puts' in binary mode. If there is a way to get it in ascii mode, that would simplify things.
– canofwhoopass
Nov 11 at 20:43












Quite opposite, retrieving a file and putting it back in the binary mode cannot change the file in any way. That's the principle of the binary mode. Moreover, there's no difference between using put and putfo in this respect. Actually all that put does is that it calls open and putfo, just like your code does.
– Martin Prikryl
Nov 11 at 22:39






Quite opposite, retrieving a file and putting it back in the binary mode cannot change the file in any way. That's the principle of the binary mode. Moreover, there's no difference between using put and putfo in this respect. Actually all that put does is that it calls open and putfo, just like your code does.
– Martin Prikryl
Nov 11 at 22:39














Thanks for the tip - complete novice here at this end. Didn't know how to explain why my new file had ^M markers when the originals didn't if the real SW guys asked.
– canofwhoopass
Nov 12 at 2:54






Thanks for the tip - complete novice here at this end. Didn't know how to explain why my new file had ^M markers when the originals didn't if the real SW guys asked.
– canofwhoopass
Nov 12 at 2:54














1 Answer
1






active

oldest

votes

















up vote
0
down vote



accepted










If you want to read the file second time, you have to seek file read pointer back to the file beginning:



with sftp_client.open(file_path) as f:
sftp_client.putfo(f, backup_path)
f.seek(0, 0)
settings = json.load(f)


Though that is functionally equivalent to your original code with two open's.





If you aim was to optimize the code, to avoid downloading the file twice, you will have to read/cache the file to memory and then upload and load the contents from the cache.



f = BytesIO()
sftp_client.getfo(file_path, f)
f.seek(0, 0)
sftp_client.putfo(f, backup_path)
f.seek(0, 0)
settings = json.load(f)





share|improve this answer





















  • f.seek worked like a charm! Thanks a bunch Martin; reading up on file.seek to educate myself.
    – canofwhoopass
    Nov 12 at 2:50











Your Answer






StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");

StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});

function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});


}
});














draft saved

draft discarded


















StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53252210%2fparamiko-download-process-and-re-upload-the-same-file%23new-answer', 'question_page');
}
);

Post as a guest















Required, but never shown

























1 Answer
1






active

oldest

votes








1 Answer
1






active

oldest

votes









active

oldest

votes






active

oldest

votes








up vote
0
down vote



accepted










If you want to read the file second time, you have to seek file read pointer back to the file beginning:



with sftp_client.open(file_path) as f:
sftp_client.putfo(f, backup_path)
f.seek(0, 0)
settings = json.load(f)


Though that is functionally equivalent to your original code with two open's.





If you aim was to optimize the code, to avoid downloading the file twice, you will have to read/cache the file to memory and then upload and load the contents from the cache.



f = BytesIO()
sftp_client.getfo(file_path, f)
f.seek(0, 0)
sftp_client.putfo(f, backup_path)
f.seek(0, 0)
settings = json.load(f)





share|improve this answer





















  • f.seek worked like a charm! Thanks a bunch Martin; reading up on file.seek to educate myself.
    – canofwhoopass
    Nov 12 at 2:50















up vote
0
down vote



accepted










If you want to read the file second time, you have to seek file read pointer back to the file beginning:



with sftp_client.open(file_path) as f:
sftp_client.putfo(f, backup_path)
f.seek(0, 0)
settings = json.load(f)


Though that is functionally equivalent to your original code with two open's.





If you aim was to optimize the code, to avoid downloading the file twice, you will have to read/cache the file to memory and then upload and load the contents from the cache.



f = BytesIO()
sftp_client.getfo(file_path, f)
f.seek(0, 0)
sftp_client.putfo(f, backup_path)
f.seek(0, 0)
settings = json.load(f)





share|improve this answer





















  • f.seek worked like a charm! Thanks a bunch Martin; reading up on file.seek to educate myself.
    – canofwhoopass
    Nov 12 at 2:50













up vote
0
down vote



accepted







up vote
0
down vote



accepted






If you want to read the file second time, you have to seek file read pointer back to the file beginning:



with sftp_client.open(file_path) as f:
sftp_client.putfo(f, backup_path)
f.seek(0, 0)
settings = json.load(f)


Though that is functionally equivalent to your original code with two open's.





If you aim was to optimize the code, to avoid downloading the file twice, you will have to read/cache the file to memory and then upload and load the contents from the cache.



f = BytesIO()
sftp_client.getfo(file_path, f)
f.seek(0, 0)
sftp_client.putfo(f, backup_path)
f.seek(0, 0)
settings = json.load(f)





share|improve this answer












If you want to read the file second time, you have to seek file read pointer back to the file beginning:



with sftp_client.open(file_path) as f:
sftp_client.putfo(f, backup_path)
f.seek(0, 0)
settings = json.load(f)


Though that is functionally equivalent to your original code with two open's.





If you aim was to optimize the code, to avoid downloading the file twice, you will have to read/cache the file to memory and then upload and load the contents from the cache.



f = BytesIO()
sftp_client.getfo(file_path, f)
f.seek(0, 0)
sftp_client.putfo(f, backup_path)
f.seek(0, 0)
settings = json.load(f)






share|improve this answer












share|improve this answer



share|improve this answer










answered Nov 11 at 19:53









Martin Prikryl

84k22159351




84k22159351












  • f.seek worked like a charm! Thanks a bunch Martin; reading up on file.seek to educate myself.
    – canofwhoopass
    Nov 12 at 2:50


















  • f.seek worked like a charm! Thanks a bunch Martin; reading up on file.seek to educate myself.
    – canofwhoopass
    Nov 12 at 2:50
















f.seek worked like a charm! Thanks a bunch Martin; reading up on file.seek to educate myself.
– canofwhoopass
Nov 12 at 2:50




f.seek worked like a charm! Thanks a bunch Martin; reading up on file.seek to educate myself.
– canofwhoopass
Nov 12 at 2:50


















draft saved

draft discarded




















































Thanks for contributing an answer to Stack Overflow!


  • Please be sure to answer the question. Provide details and share your research!

But avoid



  • Asking for help, clarification, or responding to other answers.

  • Making statements based on opinion; back them up with references or personal experience.


To learn more, see our tips on writing great answers.





Some of your past answers have not been well-received, and you're in danger of being blocked from answering.


Please pay close attention to the following guidance:


  • Please be sure to answer the question. Provide details and share your research!

But avoid



  • Asking for help, clarification, or responding to other answers.

  • Making statements based on opinion; back them up with references or personal experience.


To learn more, see our tips on writing great answers.




draft saved


draft discarded














StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53252210%2fparamiko-download-process-and-re-upload-the-same-file%23new-answer', 'question_page');
}
);

Post as a guest















Required, but never shown





















































Required, but never shown














Required, but never shown












Required, but never shown







Required, but never shown

































Required, but never shown














Required, but never shown












Required, but never shown







Required, but never shown







Popular posts from this blog

Guess what letter conforming each word

Port of Spain

Run scheduled task as local user group (not BUILTIN)