Paramiko Download, process and re-upload the same file
up vote
1
down vote
favorite
I am using Paramiko to create an SFTP client to create a backup copy of a JSON file, read in the contents of the original, then update (the original). I am able to get this snippet of code to work:
# open sftp connection stuff
# read in json create backup copy - but have to 'open' twice
read_file = sftp_client.open(file_path)
settings = json.load(read_file)
read_file = sftp_client.open(file_path)
sftp_client.putfo(read_file, backup_path)
# json stuff and updating
new_settings = json.dumps(settings, indent=4, sort_keys = True)
# update remote json file
with sftp_client.open(file_path, 'w') as f:
f.write(new_settings)
However when I try to clean up the code and combine the backup file creation and JSON load:
with sftp_client.open(file_path) as f:
sftp_client.putfo(f, backup_path)
settings = json.load(f)
The backup file will be created but json.load
will fail to due not having any content. And if I reverse the order, the json.load
will read in the values, but the backup copy will be empty.
I'm using Python 2.7 on a Windows machine, creating a remote connection to a QNX (Linux) machine. Appreciate any help.
Thanks in advance.
python sftp paramiko
add a comment |
up vote
1
down vote
favorite
I am using Paramiko to create an SFTP client to create a backup copy of a JSON file, read in the contents of the original, then update (the original). I am able to get this snippet of code to work:
# open sftp connection stuff
# read in json create backup copy - but have to 'open' twice
read_file = sftp_client.open(file_path)
settings = json.load(read_file)
read_file = sftp_client.open(file_path)
sftp_client.putfo(read_file, backup_path)
# json stuff and updating
new_settings = json.dumps(settings, indent=4, sort_keys = True)
# update remote json file
with sftp_client.open(file_path, 'w') as f:
f.write(new_settings)
However when I try to clean up the code and combine the backup file creation and JSON load:
with sftp_client.open(file_path) as f:
sftp_client.putfo(f, backup_path)
settings = json.load(f)
The backup file will be created but json.load
will fail to due not having any content. And if I reverse the order, the json.load
will read in the values, but the backup copy will be empty.
I'm using Python 2.7 on a Windows machine, creating a remote connection to a QNX (Linux) machine. Appreciate any help.
Thanks in advance.
python sftp paramiko
"I am avoiding the 'put' command to preserve the ascii/text format of the original file." - That makes no sense.
– Martin Prikryl
Nov 11 at 20:00
Retrieving and the 'putting' the file back as a binary puts a lot of ctrl-m characters at the end of lines. I'd prefer not to have to search and replace in VI to remove them. Although I am not sure if ^M will mess up the processes that read in the json file in question. Just wanted to be safe...
– canofwhoopass
Nov 11 at 20:35
Also want to add that my limited understanding is that paramiko only 'puts' in binary mode. If there is a way to get it in ascii mode, that would simplify things.
– canofwhoopass
Nov 11 at 20:43
Quite opposite, retrieving a file and putting it back in the binary mode cannot change the file in any way. That's the principle of the binary mode. Moreover, there's no difference between usingput
andputfo
in this respect. Actually all thatput
does is that it callsopen
andputfo
, just like your code does.
– Martin Prikryl
Nov 11 at 22:39
Thanks for the tip - complete novice here at this end. Didn't know how to explain why my new file had ^M markers when the originals didn't if the real SW guys asked.
– canofwhoopass
Nov 12 at 2:54
add a comment |
up vote
1
down vote
favorite
up vote
1
down vote
favorite
I am using Paramiko to create an SFTP client to create a backup copy of a JSON file, read in the contents of the original, then update (the original). I am able to get this snippet of code to work:
# open sftp connection stuff
# read in json create backup copy - but have to 'open' twice
read_file = sftp_client.open(file_path)
settings = json.load(read_file)
read_file = sftp_client.open(file_path)
sftp_client.putfo(read_file, backup_path)
# json stuff and updating
new_settings = json.dumps(settings, indent=4, sort_keys = True)
# update remote json file
with sftp_client.open(file_path, 'w') as f:
f.write(new_settings)
However when I try to clean up the code and combine the backup file creation and JSON load:
with sftp_client.open(file_path) as f:
sftp_client.putfo(f, backup_path)
settings = json.load(f)
The backup file will be created but json.load
will fail to due not having any content. And if I reverse the order, the json.load
will read in the values, but the backup copy will be empty.
I'm using Python 2.7 on a Windows machine, creating a remote connection to a QNX (Linux) machine. Appreciate any help.
Thanks in advance.
python sftp paramiko
I am using Paramiko to create an SFTP client to create a backup copy of a JSON file, read in the contents of the original, then update (the original). I am able to get this snippet of code to work:
# open sftp connection stuff
# read in json create backup copy - but have to 'open' twice
read_file = sftp_client.open(file_path)
settings = json.load(read_file)
read_file = sftp_client.open(file_path)
sftp_client.putfo(read_file, backup_path)
# json stuff and updating
new_settings = json.dumps(settings, indent=4, sort_keys = True)
# update remote json file
with sftp_client.open(file_path, 'w') as f:
f.write(new_settings)
However when I try to clean up the code and combine the backup file creation and JSON load:
with sftp_client.open(file_path) as f:
sftp_client.putfo(f, backup_path)
settings = json.load(f)
The backup file will be created but json.load
will fail to due not having any content. And if I reverse the order, the json.load
will read in the values, but the backup copy will be empty.
I'm using Python 2.7 on a Windows machine, creating a remote connection to a QNX (Linux) machine. Appreciate any help.
Thanks in advance.
python sftp paramiko
python sftp paramiko
edited Nov 12 at 7:00
Martin Prikryl
84k22159351
84k22159351
asked Nov 11 at 19:09
canofwhoopass
155
155
"I am avoiding the 'put' command to preserve the ascii/text format of the original file." - That makes no sense.
– Martin Prikryl
Nov 11 at 20:00
Retrieving and the 'putting' the file back as a binary puts a lot of ctrl-m characters at the end of lines. I'd prefer not to have to search and replace in VI to remove them. Although I am not sure if ^M will mess up the processes that read in the json file in question. Just wanted to be safe...
– canofwhoopass
Nov 11 at 20:35
Also want to add that my limited understanding is that paramiko only 'puts' in binary mode. If there is a way to get it in ascii mode, that would simplify things.
– canofwhoopass
Nov 11 at 20:43
Quite opposite, retrieving a file and putting it back in the binary mode cannot change the file in any way. That's the principle of the binary mode. Moreover, there's no difference between usingput
andputfo
in this respect. Actually all thatput
does is that it callsopen
andputfo
, just like your code does.
– Martin Prikryl
Nov 11 at 22:39
Thanks for the tip - complete novice here at this end. Didn't know how to explain why my new file had ^M markers when the originals didn't if the real SW guys asked.
– canofwhoopass
Nov 12 at 2:54
add a comment |
"I am avoiding the 'put' command to preserve the ascii/text format of the original file." - That makes no sense.
– Martin Prikryl
Nov 11 at 20:00
Retrieving and the 'putting' the file back as a binary puts a lot of ctrl-m characters at the end of lines. I'd prefer not to have to search and replace in VI to remove them. Although I am not sure if ^M will mess up the processes that read in the json file in question. Just wanted to be safe...
– canofwhoopass
Nov 11 at 20:35
Also want to add that my limited understanding is that paramiko only 'puts' in binary mode. If there is a way to get it in ascii mode, that would simplify things.
– canofwhoopass
Nov 11 at 20:43
Quite opposite, retrieving a file and putting it back in the binary mode cannot change the file in any way. That's the principle of the binary mode. Moreover, there's no difference between usingput
andputfo
in this respect. Actually all thatput
does is that it callsopen
andputfo
, just like your code does.
– Martin Prikryl
Nov 11 at 22:39
Thanks for the tip - complete novice here at this end. Didn't know how to explain why my new file had ^M markers when the originals didn't if the real SW guys asked.
– canofwhoopass
Nov 12 at 2:54
"I am avoiding the 'put' command to preserve the ascii/text format of the original file." - That makes no sense.
– Martin Prikryl
Nov 11 at 20:00
"I am avoiding the 'put' command to preserve the ascii/text format of the original file." - That makes no sense.
– Martin Prikryl
Nov 11 at 20:00
Retrieving and the 'putting' the file back as a binary puts a lot of ctrl-m characters at the end of lines. I'd prefer not to have to search and replace in VI to remove them. Although I am not sure if ^M will mess up the processes that read in the json file in question. Just wanted to be safe...
– canofwhoopass
Nov 11 at 20:35
Retrieving and the 'putting' the file back as a binary puts a lot of ctrl-m characters at the end of lines. I'd prefer not to have to search and replace in VI to remove them. Although I am not sure if ^M will mess up the processes that read in the json file in question. Just wanted to be safe...
– canofwhoopass
Nov 11 at 20:35
Also want to add that my limited understanding is that paramiko only 'puts' in binary mode. If there is a way to get it in ascii mode, that would simplify things.
– canofwhoopass
Nov 11 at 20:43
Also want to add that my limited understanding is that paramiko only 'puts' in binary mode. If there is a way to get it in ascii mode, that would simplify things.
– canofwhoopass
Nov 11 at 20:43
Quite opposite, retrieving a file and putting it back in the binary mode cannot change the file in any way. That's the principle of the binary mode. Moreover, there's no difference between using
put
and putfo
in this respect. Actually all that put
does is that it calls open
and putfo
, just like your code does.– Martin Prikryl
Nov 11 at 22:39
Quite opposite, retrieving a file and putting it back in the binary mode cannot change the file in any way. That's the principle of the binary mode. Moreover, there's no difference between using
put
and putfo
in this respect. Actually all that put
does is that it calls open
and putfo
, just like your code does.– Martin Prikryl
Nov 11 at 22:39
Thanks for the tip - complete novice here at this end. Didn't know how to explain why my new file had ^M markers when the originals didn't if the real SW guys asked.
– canofwhoopass
Nov 12 at 2:54
Thanks for the tip - complete novice here at this end. Didn't know how to explain why my new file had ^M markers when the originals didn't if the real SW guys asked.
– canofwhoopass
Nov 12 at 2:54
add a comment |
1 Answer
1
active
oldest
votes
up vote
0
down vote
accepted
If you want to read the file second time, you have to seek file read pointer back to the file beginning:
with sftp_client.open(file_path) as f:
sftp_client.putfo(f, backup_path)
f.seek(0, 0)
settings = json.load(f)
Though that is functionally equivalent to your original code with two open
's.
If you aim was to optimize the code, to avoid downloading the file twice, you will have to read/cache the file to memory and then upload and load the contents from the cache.
f = BytesIO()
sftp_client.getfo(file_path, f)
f.seek(0, 0)
sftp_client.putfo(f, backup_path)
f.seek(0, 0)
settings = json.load(f)
f.seek worked like a charm! Thanks a bunch Martin; reading up on file.seek to educate myself.
– canofwhoopass
Nov 12 at 2:50
add a comment |
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
up vote
0
down vote
accepted
If you want to read the file second time, you have to seek file read pointer back to the file beginning:
with sftp_client.open(file_path) as f:
sftp_client.putfo(f, backup_path)
f.seek(0, 0)
settings = json.load(f)
Though that is functionally equivalent to your original code with two open
's.
If you aim was to optimize the code, to avoid downloading the file twice, you will have to read/cache the file to memory and then upload and load the contents from the cache.
f = BytesIO()
sftp_client.getfo(file_path, f)
f.seek(0, 0)
sftp_client.putfo(f, backup_path)
f.seek(0, 0)
settings = json.load(f)
f.seek worked like a charm! Thanks a bunch Martin; reading up on file.seek to educate myself.
– canofwhoopass
Nov 12 at 2:50
add a comment |
up vote
0
down vote
accepted
If you want to read the file second time, you have to seek file read pointer back to the file beginning:
with sftp_client.open(file_path) as f:
sftp_client.putfo(f, backup_path)
f.seek(0, 0)
settings = json.load(f)
Though that is functionally equivalent to your original code with two open
's.
If you aim was to optimize the code, to avoid downloading the file twice, you will have to read/cache the file to memory and then upload and load the contents from the cache.
f = BytesIO()
sftp_client.getfo(file_path, f)
f.seek(0, 0)
sftp_client.putfo(f, backup_path)
f.seek(0, 0)
settings = json.load(f)
f.seek worked like a charm! Thanks a bunch Martin; reading up on file.seek to educate myself.
– canofwhoopass
Nov 12 at 2:50
add a comment |
up vote
0
down vote
accepted
up vote
0
down vote
accepted
If you want to read the file second time, you have to seek file read pointer back to the file beginning:
with sftp_client.open(file_path) as f:
sftp_client.putfo(f, backup_path)
f.seek(0, 0)
settings = json.load(f)
Though that is functionally equivalent to your original code with two open
's.
If you aim was to optimize the code, to avoid downloading the file twice, you will have to read/cache the file to memory and then upload and load the contents from the cache.
f = BytesIO()
sftp_client.getfo(file_path, f)
f.seek(0, 0)
sftp_client.putfo(f, backup_path)
f.seek(0, 0)
settings = json.load(f)
If you want to read the file second time, you have to seek file read pointer back to the file beginning:
with sftp_client.open(file_path) as f:
sftp_client.putfo(f, backup_path)
f.seek(0, 0)
settings = json.load(f)
Though that is functionally equivalent to your original code with two open
's.
If you aim was to optimize the code, to avoid downloading the file twice, you will have to read/cache the file to memory and then upload and load the contents from the cache.
f = BytesIO()
sftp_client.getfo(file_path, f)
f.seek(0, 0)
sftp_client.putfo(f, backup_path)
f.seek(0, 0)
settings = json.load(f)
answered Nov 11 at 19:53
Martin Prikryl
84k22159351
84k22159351
f.seek worked like a charm! Thanks a bunch Martin; reading up on file.seek to educate myself.
– canofwhoopass
Nov 12 at 2:50
add a comment |
f.seek worked like a charm! Thanks a bunch Martin; reading up on file.seek to educate myself.
– canofwhoopass
Nov 12 at 2:50
f.seek worked like a charm! Thanks a bunch Martin; reading up on file.seek to educate myself.
– canofwhoopass
Nov 12 at 2:50
f.seek worked like a charm! Thanks a bunch Martin; reading up on file.seek to educate myself.
– canofwhoopass
Nov 12 at 2:50
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Some of your past answers have not been well-received, and you're in danger of being blocked from answering.
Please pay close attention to the following guidance:
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53252210%2fparamiko-download-process-and-re-upload-the-same-file%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
"I am avoiding the 'put' command to preserve the ascii/text format of the original file." - That makes no sense.
– Martin Prikryl
Nov 11 at 20:00
Retrieving and the 'putting' the file back as a binary puts a lot of ctrl-m characters at the end of lines. I'd prefer not to have to search and replace in VI to remove them. Although I am not sure if ^M will mess up the processes that read in the json file in question. Just wanted to be safe...
– canofwhoopass
Nov 11 at 20:35
Also want to add that my limited understanding is that paramiko only 'puts' in binary mode. If there is a way to get it in ascii mode, that would simplify things.
– canofwhoopass
Nov 11 at 20:43
Quite opposite, retrieving a file and putting it back in the binary mode cannot change the file in any way. That's the principle of the binary mode. Moreover, there's no difference between using
put
andputfo
in this respect. Actually all thatput
does is that it callsopen
andputfo
, just like your code does.– Martin Prikryl
Nov 11 at 22:39
Thanks for the tip - complete novice here at this end. Didn't know how to explain why my new file had ^M markers when the originals didn't if the real SW guys asked.
– canofwhoopass
Nov 12 at 2:54