Evaluate a condition BEFORE put() in NDB and GAE
I have a model in my GAE based application, which is being updated quite frequently. And in some cases, it happens that same entity is being updated nearly at the same time. The entity update functionality in the app works like that:
- User enters an ID and other properties of the entity which are to be updated.
- Retrieve entity from DB against this ID (to make sure ID was valid).
- Do bunch of validation on the properties which are going to be updated (e.g. if group_id property is being updated then make sure group is present in DB against this ID and ID is integer.)
- After validations, call put() on the entity which was retrieved at step #2.
As, i mentioned same entity can be updated multiple times nearly at the same time, so i came across the classical race condition issue. i.e. let's say 2 update calls were made sequentially, in first call entity was retrieved and validations were in process but, at the same time second call triggered it retrieves the same and updates it properties. The second call also executes put() and update entity in DB. NOTE that first call was not completed yet (due to some time delay), it completes now and calls put() and updates entity in DB.
The end result in DB will be off first update call, but expected result was of second call!
I researched on GAE about this and found pre-put hooks. I think i can use "updated" timestamps to solve this issue i.e. to make sure second call only updates an entity after first call does it. But, i want to use some better approach here e.g. some DB's (like in AWS) provide tags with each DB row and we can ask DB itself to validate this tag before actually putting the record. I am curious as is there any way in GAE to do so. i.e. to ask DB to do conditional put() instead of using pre_put hooks manually?
google-app-engine google-cloud-datastore app-engine-ndb google-app-engine-python
asked Nov 17 '18 at 11:07
HamidArrivyHamidArrivy
5219
add a comment |
I have a model in my GAE based application, which is being updated quite frequently. And in some cases, it happens that same entity is being updated nearly at the same time. The entity update functionality in the app works like that:
- User enters an ID and other properties of the entity which are to be updated.
- Retrieve entity from DB against this ID (to make sure ID was valid).
- Do bunch of validation on the properties which are going to be updated (e.g. if group_id property is being updated then make sure group is present in DB against this ID and ID is integer.)
- After validations, call put() on the entity which was retrieved at step #2.
As, i mentioned same entity can be updated multiple times nearly at the same time, so i came across the classical race condition issue. i.e. let's say 2 update calls were made sequentially, in first call entity was retrieved and validations were in process but, at the same time second call triggered it retrieves the same and updates it properties. The second call also executes put() and update entity in DB. NOTE that first call was not completed yet (due to some time delay), it completes now and calls put() and updates entity in DB.
The end result in DB will be off first update call, but expected result was of second call!
I researched on GAE about this and found pre-put hooks. I think i can use "updated" timestamps to solve this issue i.e. to make sure second call only updates an entity after first call does it. But, i want to use some better approach here e.g. some DB's (like in AWS) provide tags with each DB row and we can ask DB itself to validate this tag before actually putting the record. I am curious as is there any way in GAE to do so. i.e. to ask DB to do conditional put() instead of using pre_put hooks manually?
google-app-engine google-cloud-datastore app-engine-ndb google-app-engine-python
asked Nov 17 '18 at 11:07
HamidArrivyHamidArrivy
5219
Check out transactions in GAE. This is what you need.
– Jeff O'Neill
Nov 17 '18 at 13:31
@JeffO'Neill I checked it already and learned that it is using "Optimistic Concurrency Control". You are right, this is what i need. But, transactions has their own limitations which i would like to avoid. That's why i am looking for doing it in a way other than transactions.
– HamidArrivy
Nov 17 '18 at 13:58
Maybe use the SQL database instead of datastore. Implementing your own locking mechanism is not an easy thing to do...
– Jeff O'Neill
Nov 17 '18 at 19:05
add a comment |
I have a model in my GAE based application, which is being updated quite frequently. And in some cases, it happens that same entity is being updated nearly at the same time. The entity update functionality in the app works like that:
- User enters an ID and other properties of the entity which are to be updated.
- Retrieve entity from DB against this ID (to make sure ID was valid).
- Do bunch of validation on the properties which are going to be updated (e.g. if group_id property is being updated then make sure group is present in DB against this ID and ID is integer.)
- After validations, call put() on the entity which was retrieved at step #2.
As, i mentioned same entity can be updated multiple times nearly at the same time, so i came across the classical race condition issue. i.e. let's say 2 update calls were made sequentially, in first call entity was retrieved and validations were in process but, at the same time second call triggered it retrieves the same and updates it properties. The second call also executes put() and update entity in DB. NOTE that first call was not completed yet (due to some time delay), it completes now and calls put() and updates entity in DB.
The end result in DB will be off first update call, but expected result was of second call!
I researched on GAE about this and found pre-put hooks. I think i can use "updated" timestamps to solve this issue i.e. to make sure second call only updates an entity after first call does it. But, i want to use some better approach here e.g. some DB's (like in AWS) provide tags with each DB row and we can ask DB itself to validate this tag before actually putting the record. I am curious as is there any way in GAE to do so. i.e. to ask DB to do conditional put() instead of using pre_put hooks manually?
google-app-engine google-cloud-datastore app-engine-ndb google-app-engine-python
asked Nov 17 '18 at 11:07
HamidArrivyHamidArrivy
5219
I have a model in my GAE based application, which is being updated quite frequently. And in some cases, it happens that same entity is being updated nearly at the same time. The entity update functionality in the app works like that:
- User enters an ID and other properties of the entity which are to be updated.
- Retrieve entity from DB against this ID (to make sure ID was valid).
- Do bunch of validation on the properties which are going to be updated (e.g. if group_id property is being updated then make sure group is present in DB against this ID and ID is integer.)
- After validations, call put() on the entity which was retrieved at step #2.
As, i mentioned same entity can be updated multiple times nearly at the same time, so i came across the classical race condition issue. i.e. let's say 2 update calls were made sequentially, in first call entity was retrieved and validations were in process but, at the same time second call triggered it retrieves the same and updates it properties. The second call also executes put() and update entity in DB. NOTE that first call was not completed yet (due to some time delay), it completes now and calls put() and updates entity in DB.
The end result in DB will be off first update call, but expected result was of second call!
I researched on GAE about this and found pre-put hooks. I think i can use "updated" timestamps to solve this issue i.e. to make sure second call only updates an entity after first call does it. But, i want to use some better approach here e.g. some DB's (like in AWS) provide tags with each DB row and we can ask DB itself to validate this tag before actually putting the record. I am curious as is there any way in GAE to do so. i.e. to ask DB to do conditional put() instead of using pre_put hooks manually?
google-app-engine google-cloud-datastore app-engine-ndb google-app-engine-python
google-app-engine google-cloud-datastore app-engine-ndb google-app-engine-python
asked Nov 17 '18 at 11:07
HamidArrivyHamidArrivy
5219
asked Nov 17 '18 at 11:07
HamidArrivyHamidArrivy
5219
edited Nov 17 '18 at 12:08
HamidArrivy
asked Nov 17 '18 at 11:07
HamidArrivyHamidArrivy
5219
asked Nov 17 '18 at 11:07
HamidArrivyHamidArrivy
5219
asked Nov 17 '18 at 11:07
HamidArrivyHamidArrivy
5219
5219
Check out transactions in GAE. This is what you need.
– Jeff O'Neill
Nov 17 '18 at 13:31
@JeffO'Neill I checked it already and learned that it is using "Optimistic Concurrency Control". You are right, this is what i need. But, transactions has their own limitations which i would like to avoid. That's why i am looking for doing it in a way other than transactions.
– HamidArrivy
Nov 17 '18 at 13:58
Maybe use the SQL database instead of datastore. Implementing your own locking mechanism is not an easy thing to do...
– Jeff O'Neill
Nov 17 '18 at 19:05
add a comment |
Check out transactions in GAE. This is what you need.
– Jeff O'Neill
Nov 17 '18 at 13:31
@JeffO'Neill I checked it already and learned that it is using "Optimistic Concurrency Control". You are right, this is what i need. But, transactions has their own limitations which i would like to avoid. That's why i am looking for doing it in a way other than transactions.
– HamidArrivy
Nov 17 '18 at 13:58
Maybe use the SQL database instead of datastore. Implementing your own locking mechanism is not an easy thing to do...
– Jeff O'Neill
Nov 17 '18 at 19:05
Check out transactions in GAE. This is what you need.
– Jeff O'Neill
Nov 17 '18 at 13:31
Check out transactions in GAE. This is what you need.
– Jeff O'Neill
Nov 17 '18 at 13:31
@JeffO'Neill I checked it already and learned that it is using "Optimistic Concurrency Control". You are right, this is what i need. But, transactions has their own limitations which i would like to avoid. That's why i am looking for doing it in a way other than transactions.
– HamidArrivy
Nov 17 '18 at 13:58
@JeffO'Neill I checked it already and learned that it is using "Optimistic Concurrency Control". You are right, this is what i need. But, transactions has their own limitations which i would like to avoid. That's why i am looking for doing it in a way other than transactions.
– HamidArrivy
Nov 17 '18 at 13:58
Maybe use the SQL database instead of datastore. Implementing your own locking mechanism is not an easy thing to do...
– Jeff O'Neill
Nov 17 '18 at 19:05
Maybe use the SQL database instead of datastore. Implementing your own locking mechanism is not an easy thing to do...
– Jeff O'Neill
Nov 17 '18 at 19:05
add a comment |
0
active
oldest
votes
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53350606%2fevaluate-a-condition-before-put-in-ndb-and-gae%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
0
active
oldest
votes
0
active
oldest
votes
active
oldest
votes
active
oldest
votes
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53350606%2fevaluate-a-condition-before-put-in-ndb-and-gae%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Check out transactions in GAE. This is what you need.
– Jeff O'Neill
Nov 17 '18 at 13:31
@JeffO'Neill I checked it already and learned that it is using "Optimistic Concurrency Control". You are right, this is what i need. But, transactions has their own limitations which i would like to avoid. That's why i am looking for doing it in a way other than transactions.
– HamidArrivy
Nov 17 '18 at 13:58
Maybe use the SQL database instead of datastore. Implementing your own locking mechanism is not an easy thing to do...
– Jeff O'Neill
Nov 17 '18 at 19:05