S3 Eventual consistency: Is it safe to delete an object while there's an outstanding copy for it?












1















I'm issuing an S3 CopyObject (aka PUT copy) from a source bucket that is different than the destination bucket. I'm wondering if it's safe to delete the source after CopyObject has returned OK to the REST client. By "safe", I mean that the destination object will eventually show up, and that it will initially contain all of the data available at the time the copy was issued.



Corruption in the copy is perhaps unlikely (given that most operations are atomic), but the two operations cancelling each other might be possible. I wish the docs were slightly more fleshed out wrt to eventual consistency.



(in my scenario, nothing's writing to the source nor destination key in the interim. and the same client does the copy and delete).



e.g. synchronous pseudocode:



try:

# make sure this is a create. read-after-create consistency
my_tmpname = "new_tempfile" + uuid4()
s3_put(data, "s3://my-bucket1/" + my_tmpname)

...

# copy it to its final location
s3_copy("s3://my-bucket1/new_tempfile", "s3://my-bucket2/final_location")
finally:
# Cleanup temp file.
#
# Can this delete interfere with the copy in flight?
# e.g. Should one wait a few seconds/minutes?
# e.g. Should one ensure that the target exists before deleting source?
s3_delete("s3://my-bucket/new_tempfile")









share|improve this question

























  • its a synchronous call. As long as you do not delete source files from outside the script while copy is in-progress, you will be fine.

    – Asdfg
    Nov 16 '18 at 14:14


















1















I'm issuing an S3 CopyObject (aka PUT copy) from a source bucket that is different than the destination bucket. I'm wondering if it's safe to delete the source after CopyObject has returned OK to the REST client. By "safe", I mean that the destination object will eventually show up, and that it will initially contain all of the data available at the time the copy was issued.



Corruption in the copy is perhaps unlikely (given that most operations are atomic), but the two operations cancelling each other might be possible. I wish the docs were slightly more fleshed out wrt to eventual consistency.



(in my scenario, nothing's writing to the source nor destination key in the interim. and the same client does the copy and delete).



e.g. synchronous pseudocode:



try:

# make sure this is a create. read-after-create consistency
my_tmpname = "new_tempfile" + uuid4()
s3_put(data, "s3://my-bucket1/" + my_tmpname)

...

# copy it to its final location
s3_copy("s3://my-bucket1/new_tempfile", "s3://my-bucket2/final_location")
finally:
# Cleanup temp file.
#
# Can this delete interfere with the copy in flight?
# e.g. Should one wait a few seconds/minutes?
# e.g. Should one ensure that the target exists before deleting source?
s3_delete("s3://my-bucket/new_tempfile")









share|improve this question

























  • its a synchronous call. As long as you do not delete source files from outside the script while copy is in-progress, you will be fine.

    – Asdfg
    Nov 16 '18 at 14:14
















1












1








1


1






I'm issuing an S3 CopyObject (aka PUT copy) from a source bucket that is different than the destination bucket. I'm wondering if it's safe to delete the source after CopyObject has returned OK to the REST client. By "safe", I mean that the destination object will eventually show up, and that it will initially contain all of the data available at the time the copy was issued.



Corruption in the copy is perhaps unlikely (given that most operations are atomic), but the two operations cancelling each other might be possible. I wish the docs were slightly more fleshed out wrt to eventual consistency.



(in my scenario, nothing's writing to the source nor destination key in the interim. and the same client does the copy and delete).



e.g. synchronous pseudocode:



try:

# make sure this is a create. read-after-create consistency
my_tmpname = "new_tempfile" + uuid4()
s3_put(data, "s3://my-bucket1/" + my_tmpname)

...

# copy it to its final location
s3_copy("s3://my-bucket1/new_tempfile", "s3://my-bucket2/final_location")
finally:
# Cleanup temp file.
#
# Can this delete interfere with the copy in flight?
# e.g. Should one wait a few seconds/minutes?
# e.g. Should one ensure that the target exists before deleting source?
s3_delete("s3://my-bucket/new_tempfile")









share|improve this question
















I'm issuing an S3 CopyObject (aka PUT copy) from a source bucket that is different than the destination bucket. I'm wondering if it's safe to delete the source after CopyObject has returned OK to the REST client. By "safe", I mean that the destination object will eventually show up, and that it will initially contain all of the data available at the time the copy was issued.



Corruption in the copy is perhaps unlikely (given that most operations are atomic), but the two operations cancelling each other might be possible. I wish the docs were slightly more fleshed out wrt to eventual consistency.



(in my scenario, nothing's writing to the source nor destination key in the interim. and the same client does the copy and delete).



e.g. synchronous pseudocode:



try:

# make sure this is a create. read-after-create consistency
my_tmpname = "new_tempfile" + uuid4()
s3_put(data, "s3://my-bucket1/" + my_tmpname)

...

# copy it to its final location
s3_copy("s3://my-bucket1/new_tempfile", "s3://my-bucket2/final_location")
finally:
# Cleanup temp file.
#
# Can this delete interfere with the copy in flight?
# e.g. Should one wait a few seconds/minutes?
# e.g. Should one ensure that the target exists before deleting source?
s3_delete("s3://my-bucket/new_tempfile")






amazon-web-services amazon-s3






share|improve this question















share|improve this question













share|improve this question




share|improve this question








edited Nov 21 '18 at 19:59







init_js

















asked Nov 16 '18 at 1:47









init_jsinit_js

956824




956824













  • its a synchronous call. As long as you do not delete source files from outside the script while copy is in-progress, you will be fine.

    – Asdfg
    Nov 16 '18 at 14:14





















  • its a synchronous call. As long as you do not delete source files from outside the script while copy is in-progress, you will be fine.

    – Asdfg
    Nov 16 '18 at 14:14



















its a synchronous call. As long as you do not delete source files from outside the script while copy is in-progress, you will be fine.

– Asdfg
Nov 16 '18 at 14:14







its a synchronous call. As long as you do not delete source files from outside the script while copy is in-progress, you will be fine.

– Asdfg
Nov 16 '18 at 14:14














2 Answers
2






active

oldest

votes


















2














Yes, it is perfectly safe to delete an object immediately after a successful copy operation using that object as the source object, because copy operations are not asynchronous.



The copy request does not return until the operation has either succeeded or failed.




To help better ensure data durability, Amazon S3 PUT and PUT Object copy operations synchronously store your data across multiple facilities before returning SUCCESS.



https://docs.aws.amazon.com/AmazonS3/latest/dev/DataDurability.html




The consistency model in S3 is related only to the visibility of objects, not the durability of their storage.






share|improve this answer



















  • 1





    So, definitely start the delete after the copy is completed. Good to know that PUT and PUT-COPY are synchronous!

    – init_js
    Nov 16 '18 at 18:40



















0














Amazon S3 features read-after-write consistency for PUTs and the copy case you exemplified:




Amazon S3 Data Consistency Model



Amazon S3 provides read-after-write consistency for PUTS of new
objects in your S3 bucket in all regions with one caveat. The caveat
is that if you make a HEAD or GET request to the key name (to find if
the object exists) before creating the object, Amazon S3 provides
eventual consistency for read-after-write.




Check the documentation for consistency of the other operations.



https://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction.html






share|improve this answer


























  • Most of the examples on that page cover consistency done over the same key however (e.g. read-after-write of the same key).

    – init_js
    Nov 16 '18 at 18:41











Your Answer






StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");

StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});

function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});


}
});














draft saved

draft discarded


















StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53330317%2fs3-eventual-consistency-is-it-safe-to-delete-an-object-while-theres-an-outstan%23new-answer', 'question_page');
}
);

Post as a guest















Required, but never shown

























2 Answers
2






active

oldest

votes








2 Answers
2






active

oldest

votes









active

oldest

votes






active

oldest

votes









2














Yes, it is perfectly safe to delete an object immediately after a successful copy operation using that object as the source object, because copy operations are not asynchronous.



The copy request does not return until the operation has either succeeded or failed.




To help better ensure data durability, Amazon S3 PUT and PUT Object copy operations synchronously store your data across multiple facilities before returning SUCCESS.



https://docs.aws.amazon.com/AmazonS3/latest/dev/DataDurability.html




The consistency model in S3 is related only to the visibility of objects, not the durability of their storage.






share|improve this answer



















  • 1





    So, definitely start the delete after the copy is completed. Good to know that PUT and PUT-COPY are synchronous!

    – init_js
    Nov 16 '18 at 18:40
















2














Yes, it is perfectly safe to delete an object immediately after a successful copy operation using that object as the source object, because copy operations are not asynchronous.



The copy request does not return until the operation has either succeeded or failed.




To help better ensure data durability, Amazon S3 PUT and PUT Object copy operations synchronously store your data across multiple facilities before returning SUCCESS.



https://docs.aws.amazon.com/AmazonS3/latest/dev/DataDurability.html




The consistency model in S3 is related only to the visibility of objects, not the durability of their storage.






share|improve this answer



















  • 1





    So, definitely start the delete after the copy is completed. Good to know that PUT and PUT-COPY are synchronous!

    – init_js
    Nov 16 '18 at 18:40














2












2








2







Yes, it is perfectly safe to delete an object immediately after a successful copy operation using that object as the source object, because copy operations are not asynchronous.



The copy request does not return until the operation has either succeeded or failed.




To help better ensure data durability, Amazon S3 PUT and PUT Object copy operations synchronously store your data across multiple facilities before returning SUCCESS.



https://docs.aws.amazon.com/AmazonS3/latest/dev/DataDurability.html




The consistency model in S3 is related only to the visibility of objects, not the durability of their storage.






share|improve this answer













Yes, it is perfectly safe to delete an object immediately after a successful copy operation using that object as the source object, because copy operations are not asynchronous.



The copy request does not return until the operation has either succeeded or failed.




To help better ensure data durability, Amazon S3 PUT and PUT Object copy operations synchronously store your data across multiple facilities before returning SUCCESS.



https://docs.aws.amazon.com/AmazonS3/latest/dev/DataDurability.html




The consistency model in S3 is related only to the visibility of objects, not the durability of their storage.







share|improve this answer












share|improve this answer



share|improve this answer










answered Nov 16 '18 at 13:48









Michael - sqlbotMichael - sqlbot

88.4k12129191




88.4k12129191








  • 1





    So, definitely start the delete after the copy is completed. Good to know that PUT and PUT-COPY are synchronous!

    – init_js
    Nov 16 '18 at 18:40














  • 1





    So, definitely start the delete after the copy is completed. Good to know that PUT and PUT-COPY are synchronous!

    – init_js
    Nov 16 '18 at 18:40








1




1





So, definitely start the delete after the copy is completed. Good to know that PUT and PUT-COPY are synchronous!

– init_js
Nov 16 '18 at 18:40





So, definitely start the delete after the copy is completed. Good to know that PUT and PUT-COPY are synchronous!

– init_js
Nov 16 '18 at 18:40













0














Amazon S3 features read-after-write consistency for PUTs and the copy case you exemplified:




Amazon S3 Data Consistency Model



Amazon S3 provides read-after-write consistency for PUTS of new
objects in your S3 bucket in all regions with one caveat. The caveat
is that if you make a HEAD or GET request to the key name (to find if
the object exists) before creating the object, Amazon S3 provides
eventual consistency for read-after-write.




Check the documentation for consistency of the other operations.



https://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction.html






share|improve this answer


























  • Most of the examples on that page cover consistency done over the same key however (e.g. read-after-write of the same key).

    – init_js
    Nov 16 '18 at 18:41
















0














Amazon S3 features read-after-write consistency for PUTs and the copy case you exemplified:




Amazon S3 Data Consistency Model



Amazon S3 provides read-after-write consistency for PUTS of new
objects in your S3 bucket in all regions with one caveat. The caveat
is that if you make a HEAD or GET request to the key name (to find if
the object exists) before creating the object, Amazon S3 provides
eventual consistency for read-after-write.




Check the documentation for consistency of the other operations.



https://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction.html






share|improve this answer


























  • Most of the examples on that page cover consistency done over the same key however (e.g. read-after-write of the same key).

    – init_js
    Nov 16 '18 at 18:41














0












0








0







Amazon S3 features read-after-write consistency for PUTs and the copy case you exemplified:




Amazon S3 Data Consistency Model



Amazon S3 provides read-after-write consistency for PUTS of new
objects in your S3 bucket in all regions with one caveat. The caveat
is that if you make a HEAD or GET request to the key name (to find if
the object exists) before creating the object, Amazon S3 provides
eventual consistency for read-after-write.




Check the documentation for consistency of the other operations.



https://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction.html






share|improve this answer















Amazon S3 features read-after-write consistency for PUTs and the copy case you exemplified:




Amazon S3 Data Consistency Model



Amazon S3 provides read-after-write consistency for PUTS of new
objects in your S3 bucket in all regions with one caveat. The caveat
is that if you make a HEAD or GET request to the key name (to find if
the object exists) before creating the object, Amazon S3 provides
eventual consistency for read-after-write.




Check the documentation for consistency of the other operations.



https://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction.html







share|improve this answer














share|improve this answer



share|improve this answer








edited Nov 16 '18 at 13:19

























answered Nov 16 '18 at 9:42









faermanjfaermanj

8,01664565




8,01664565













  • Most of the examples on that page cover consistency done over the same key however (e.g. read-after-write of the same key).

    – init_js
    Nov 16 '18 at 18:41



















  • Most of the examples on that page cover consistency done over the same key however (e.g. read-after-write of the same key).

    – init_js
    Nov 16 '18 at 18:41

















Most of the examples on that page cover consistency done over the same key however (e.g. read-after-write of the same key).

– init_js
Nov 16 '18 at 18:41





Most of the examples on that page cover consistency done over the same key however (e.g. read-after-write of the same key).

– init_js
Nov 16 '18 at 18:41


















draft saved

draft discarded




















































Thanks for contributing an answer to Stack Overflow!


  • Please be sure to answer the question. Provide details and share your research!

But avoid



  • Asking for help, clarification, or responding to other answers.

  • Making statements based on opinion; back them up with references or personal experience.


To learn more, see our tips on writing great answers.




draft saved


draft discarded














StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53330317%2fs3-eventual-consistency-is-it-safe-to-delete-an-object-while-theres-an-outstan%23new-answer', 'question_page');
}
);

Post as a guest















Required, but never shown





















































Required, but never shown














Required, but never shown












Required, but never shown







Required, but never shown

































Required, but never shown














Required, but never shown












Required, but never shown







Required, but never shown







Popular posts from this blog

Guess what letter conforming each word

Port of Spain

Run scheduled task as local user group (not BUILTIN)