Flask Sqlalchemy compare two dates of two related tables
I have two related tables, User and UserDownload, now I want to filter out UserDownload which it's created_at is bigger than the created_at of the user plus one day, so the python code is :
result = db.session.query(UserDownload.uid).join(User, UserDownload.uid == User.id).filter(UserDownload.created_at >= User.created_at + timedelta(days=1)).all()
it's logic seems correct, but the result is weird, some result, the created time of user plus one day is little than the created_at of UserDownload, but some are not . and the raw sql I check the string of the query is :
SELECT user_downloads.uid AS user_downloads_uid nFROM user_downloads JOIN users ON user_downloads.uid = users.id nWHERE user_downloads.created_at >= users.created_at + :created_at_1
really don't know what :created_at_1 means.
for example, the result contains a such user_download(I replace UserDownload.uid with UserDownload, and query User by its uid):
user_download.created_at: datetime.datetime(2015, 12, 3, 8, 39, 56)
user.created_at: datetime.datetime(2015, 12, 2, 11, 7, 14)
python date sqlalchemy
add a comment |
I have two related tables, User and UserDownload, now I want to filter out UserDownload which it's created_at is bigger than the created_at of the user plus one day, so the python code is :
result = db.session.query(UserDownload.uid).join(User, UserDownload.uid == User.id).filter(UserDownload.created_at >= User.created_at + timedelta(days=1)).all()
it's logic seems correct, but the result is weird, some result, the created time of user plus one day is little than the created_at of UserDownload, but some are not . and the raw sql I check the string of the query is :
SELECT user_downloads.uid AS user_downloads_uid nFROM user_downloads JOIN users ON user_downloads.uid = users.id nWHERE user_downloads.created_at >= users.created_at + :created_at_1
really don't know what :created_at_1 means.
for example, the result contains a such user_download(I replace UserDownload.uid with UserDownload, and query User by its uid):
user_download.created_at: datetime.datetime(2015, 12, 3, 8, 39, 56)
user.created_at: datetime.datetime(2015, 12, 2, 11, 7, 14)
python date sqlalchemy
1
What database are you using?
– SuperShoot
Nov 20 '18 at 8:49
@SuperShoot I use Mysql as my database, and sqlalchemy as my orm in flask. SQLAlchemy==1.0.12
– FridayLi
Nov 20 '18 at 9:42
add a comment |
I have two related tables, User and UserDownload, now I want to filter out UserDownload which it's created_at is bigger than the created_at of the user plus one day, so the python code is :
result = db.session.query(UserDownload.uid).join(User, UserDownload.uid == User.id).filter(UserDownload.created_at >= User.created_at + timedelta(days=1)).all()
it's logic seems correct, but the result is weird, some result, the created time of user plus one day is little than the created_at of UserDownload, but some are not . and the raw sql I check the string of the query is :
SELECT user_downloads.uid AS user_downloads_uid nFROM user_downloads JOIN users ON user_downloads.uid = users.id nWHERE user_downloads.created_at >= users.created_at + :created_at_1
really don't know what :created_at_1 means.
for example, the result contains a such user_download(I replace UserDownload.uid with UserDownload, and query User by its uid):
user_download.created_at: datetime.datetime(2015, 12, 3, 8, 39, 56)
user.created_at: datetime.datetime(2015, 12, 2, 11, 7, 14)
python date sqlalchemy
I have two related tables, User and UserDownload, now I want to filter out UserDownload which it's created_at is bigger than the created_at of the user plus one day, so the python code is :
result = db.session.query(UserDownload.uid).join(User, UserDownload.uid == User.id).filter(UserDownload.created_at >= User.created_at + timedelta(days=1)).all()
it's logic seems correct, but the result is weird, some result, the created time of user plus one day is little than the created_at of UserDownload, but some are not . and the raw sql I check the string of the query is :
SELECT user_downloads.uid AS user_downloads_uid nFROM user_downloads JOIN users ON user_downloads.uid = users.id nWHERE user_downloads.created_at >= users.created_at + :created_at_1
really don't know what :created_at_1 means.
for example, the result contains a such user_download(I replace UserDownload.uid with UserDownload, and query User by its uid):
user_download.created_at: datetime.datetime(2015, 12, 3, 8, 39, 56)
user.created_at: datetime.datetime(2015, 12, 2, 11, 7, 14)
python date sqlalchemy
python date sqlalchemy
edited Nov 20 '18 at 8:27
FridayLi
asked Nov 20 '18 at 7:46
FridayLiFridayLi
185
185
1
What database are you using?
– SuperShoot
Nov 20 '18 at 8:49
@SuperShoot I use Mysql as my database, and sqlalchemy as my orm in flask. SQLAlchemy==1.0.12
– FridayLi
Nov 20 '18 at 9:42
add a comment |
1
What database are you using?
– SuperShoot
Nov 20 '18 at 8:49
@SuperShoot I use Mysql as my database, and sqlalchemy as my orm in flask. SQLAlchemy==1.0.12
– FridayLi
Nov 20 '18 at 9:42
1
1
What database are you using?
– SuperShoot
Nov 20 '18 at 8:49
What database are you using?
– SuperShoot
Nov 20 '18 at 8:49
@SuperShoot I use Mysql as my database, and sqlalchemy as my orm in flask. SQLAlchemy==1.0.12
– FridayLi
Nov 20 '18 at 9:42
@SuperShoot I use Mysql as my database, and sqlalchemy as my orm in flask. SQLAlchemy==1.0.12
– FridayLi
Nov 20 '18 at 9:42
add a comment |
1 Answer
1
active
oldest
votes
Depending on the backend that you are using, you need your filter to generate sql like (for mysql):
...WHERE user_downloads.created_at >= DATE_ADD(users.created_at, INTERVAL 1 DAY)
SQLAlchemy doesn't convert arithmetic between a datetime
object and an InstrumentedAttribute
object into that DATE_ADD
(or equivalent depending on backend) function. So where you filter by this:
UserDownload.created_at >= User.created_at + timedelta(days=1)
it gets converted to this:
...WHERE user_downloads.created_at >= users.created_at + :created_at_1
where it treats timedelta(days=1)
as a literal value, and parameterises it. That is what the :created_at_1
is, a parameter that holds the place in the query for the timedelta
object which will be passed along with the query to the server (as a side note, in MySQL that timedelta
actually gets converted into a datetime
object that is epoch + 1 day, as MySQL doesn't have a native INTERVAL
type).
So to get the query to do what you want, you need to use the sqlalchemy.func
object to generate the server side function that you need. Continuing with the MySQL example, an appropriate query might be:
from sqlalchemy import func, text
q = session.query(UserDownload.uid).
join(User, UserDownload.uid == User.id).
filter(
UserDownload.created_at >=
func.DATE_ADD(
User.created_at,
text('INTERVAL 1 DAY')
)
)
Which generates:
SELECT user_downloads.uid
FROM user_downloads INNER JOIN users ON user_downloads.uid = users.id
WHERE user_downloads.created_at >= DATE_ADD(users.created_at, INTERVAL 1 DAY)
I found this question helpful: Using DATEADD in sqlalchemy
From the comments
since mysql can handle timedelta(days=1) params, why the query I used
fails.
OK, I'll try to go into more detail about your original query, but give me some latitude as I'm working this out as I go. Lets forget about the timedelta
for a second and just see what the sql generated is. So this:
session.query(UserDownload.uid).join(User, UserDownload.uid == User.id).filter(UserDownload.created_at >= User.created_at)
generates this sql:
SELECT user_downloads.uid AS user_downloads_uid
FROM user_downloads INNER JOIN users ON user_downloads.uid = users.id
WHERE user_downloads.created_at >= users.created_at
Nothing hard to grok there. Now when we add back in the timedelta
the sql generated is exactly the same except that we add a value on to users.created_at
which is represented by the bind parameter created_at_1
:
SELECT user_downloads.uid AS user_downloads_uid
FROM user_downloads INNER JOIN users ON user_downloads.uid = users.id
WHERE user_downloads.created_at >= users.created_at + %(created_at_1)s
So is the comparison executed first, or the addition? Running this query...
print(engine.execute(text("SELECT 3 >= 2 + 2")).fetchall())
... returns 0
(i.e. False
), which proves that the addition (2 + 2
) is resolved before the comparison (>=
). So safe to assume that happens in your query too and from that, the value of created_at_1
is added to users.created_at
prior to the comparison with user_downloads.created_at
. When I execute your query, this is the parameter value passed to the server:
{'created_at_1': datetime.datetime(1970, 1, 2, 0, 0)}
So even though you add timedelta(days=1)
to users.created_at
in your filter, SQLAlchemy actually passes a datetime
object equivalent to <epoch> + <timedelta>
, or in this case: datetime(1970, 1, 1, 0, 0) + timedelta(days=1)
or datetime(1970, 1, 2, 0, 0)
(which is the value that you see in the parameter value dictionary above).
So what exactly is the value of user.created_at + datetime(1970, 1, 2, 0, 0)
? I added a User
instance into the database with created_at = datetime(1970, 1, 1, 0, 0)
:
session.add(User(id=1, created_at=datetime(1970, 1, 1, 0, 0)))
session.commit()
Then ran this query:
engine.execute(text("SELECT created_at + :a FROM users"), a=datetime(1970, 1, 2, 0, 0)).fetchall()
which returned:
[(19700101001970.0,)]
That is the value of user.created_at
without any formatting, with the year portion of the datetime(1970, 1, 2, 0, 0)
concatenated to the end. So that is what your query is comparing to user_download.created_at
. For the sake of (relative) brevity, I'm not going to look into how the comparison of user_download.created_at
and that value works, but hopefully I've demonstrated that the end result of your query is not comparing user_download.created_at
with users.created_at
plus 1 day.
when I use query like: User.query.filter(User.created_at >
datetime.now() + timedelta(days=1)), it works fine.
Using that query, here is the generated sql:
SELECT users.id AS users_id, users.created_at AS users_created_at
FROM users
WHERE users.created_at > %(created_at_1)s
and the value passed on to the server is:
{'created_at_1': datetime.datetime(2018, 11, 21, 21, 38, 51, 670890)}
So you can see that the datetime + timedelta
part is resolved before being passed to the database and as such, the database operation is a simple comparison of a DATETIME
column to a datetime
value.
Thanks, It helps. But I still have a question, since mysql can handle timedelta(days=1) params, why the query I used fails. Btw, when I use query like: User.query.filter(User.created_at > datetime.now() + timedelta(days=1)), it works fine. Is it because you can't do the date add between python object and the column in database?
– FridayLi
Nov 20 '18 at 9:58
I've added to the answer above to try and address your extra questions.
– SuperShoot
Nov 20 '18 at 11:46
thanks so much, you explain it very well.
– FridayLi
Nov 20 '18 at 12:17
btw, 19700101001970.0, means the original date plus 19min and 70 seconds to me. and when I migrate the original data to mongo, and query it by aggregate, it seems like the origin query filter out user_download bigger than user created_at plus about 20mins.
– FridayLi
Nov 21 '18 at 1:55
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53388374%2fflask-sqlalchemy-compare-two-dates-of-two-related-tables%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
Depending on the backend that you are using, you need your filter to generate sql like (for mysql):
...WHERE user_downloads.created_at >= DATE_ADD(users.created_at, INTERVAL 1 DAY)
SQLAlchemy doesn't convert arithmetic between a datetime
object and an InstrumentedAttribute
object into that DATE_ADD
(or equivalent depending on backend) function. So where you filter by this:
UserDownload.created_at >= User.created_at + timedelta(days=1)
it gets converted to this:
...WHERE user_downloads.created_at >= users.created_at + :created_at_1
where it treats timedelta(days=1)
as a literal value, and parameterises it. That is what the :created_at_1
is, a parameter that holds the place in the query for the timedelta
object which will be passed along with the query to the server (as a side note, in MySQL that timedelta
actually gets converted into a datetime
object that is epoch + 1 day, as MySQL doesn't have a native INTERVAL
type).
So to get the query to do what you want, you need to use the sqlalchemy.func
object to generate the server side function that you need. Continuing with the MySQL example, an appropriate query might be:
from sqlalchemy import func, text
q = session.query(UserDownload.uid).
join(User, UserDownload.uid == User.id).
filter(
UserDownload.created_at >=
func.DATE_ADD(
User.created_at,
text('INTERVAL 1 DAY')
)
)
Which generates:
SELECT user_downloads.uid
FROM user_downloads INNER JOIN users ON user_downloads.uid = users.id
WHERE user_downloads.created_at >= DATE_ADD(users.created_at, INTERVAL 1 DAY)
I found this question helpful: Using DATEADD in sqlalchemy
From the comments
since mysql can handle timedelta(days=1) params, why the query I used
fails.
OK, I'll try to go into more detail about your original query, but give me some latitude as I'm working this out as I go. Lets forget about the timedelta
for a second and just see what the sql generated is. So this:
session.query(UserDownload.uid).join(User, UserDownload.uid == User.id).filter(UserDownload.created_at >= User.created_at)
generates this sql:
SELECT user_downloads.uid AS user_downloads_uid
FROM user_downloads INNER JOIN users ON user_downloads.uid = users.id
WHERE user_downloads.created_at >= users.created_at
Nothing hard to grok there. Now when we add back in the timedelta
the sql generated is exactly the same except that we add a value on to users.created_at
which is represented by the bind parameter created_at_1
:
SELECT user_downloads.uid AS user_downloads_uid
FROM user_downloads INNER JOIN users ON user_downloads.uid = users.id
WHERE user_downloads.created_at >= users.created_at + %(created_at_1)s
So is the comparison executed first, or the addition? Running this query...
print(engine.execute(text("SELECT 3 >= 2 + 2")).fetchall())
... returns 0
(i.e. False
), which proves that the addition (2 + 2
) is resolved before the comparison (>=
). So safe to assume that happens in your query too and from that, the value of created_at_1
is added to users.created_at
prior to the comparison with user_downloads.created_at
. When I execute your query, this is the parameter value passed to the server:
{'created_at_1': datetime.datetime(1970, 1, 2, 0, 0)}
So even though you add timedelta(days=1)
to users.created_at
in your filter, SQLAlchemy actually passes a datetime
object equivalent to <epoch> + <timedelta>
, or in this case: datetime(1970, 1, 1, 0, 0) + timedelta(days=1)
or datetime(1970, 1, 2, 0, 0)
(which is the value that you see in the parameter value dictionary above).
So what exactly is the value of user.created_at + datetime(1970, 1, 2, 0, 0)
? I added a User
instance into the database with created_at = datetime(1970, 1, 1, 0, 0)
:
session.add(User(id=1, created_at=datetime(1970, 1, 1, 0, 0)))
session.commit()
Then ran this query:
engine.execute(text("SELECT created_at + :a FROM users"), a=datetime(1970, 1, 2, 0, 0)).fetchall()
which returned:
[(19700101001970.0,)]
That is the value of user.created_at
without any formatting, with the year portion of the datetime(1970, 1, 2, 0, 0)
concatenated to the end. So that is what your query is comparing to user_download.created_at
. For the sake of (relative) brevity, I'm not going to look into how the comparison of user_download.created_at
and that value works, but hopefully I've demonstrated that the end result of your query is not comparing user_download.created_at
with users.created_at
plus 1 day.
when I use query like: User.query.filter(User.created_at >
datetime.now() + timedelta(days=1)), it works fine.
Using that query, here is the generated sql:
SELECT users.id AS users_id, users.created_at AS users_created_at
FROM users
WHERE users.created_at > %(created_at_1)s
and the value passed on to the server is:
{'created_at_1': datetime.datetime(2018, 11, 21, 21, 38, 51, 670890)}
So you can see that the datetime + timedelta
part is resolved before being passed to the database and as such, the database operation is a simple comparison of a DATETIME
column to a datetime
value.
Thanks, It helps. But I still have a question, since mysql can handle timedelta(days=1) params, why the query I used fails. Btw, when I use query like: User.query.filter(User.created_at > datetime.now() + timedelta(days=1)), it works fine. Is it because you can't do the date add between python object and the column in database?
– FridayLi
Nov 20 '18 at 9:58
I've added to the answer above to try and address your extra questions.
– SuperShoot
Nov 20 '18 at 11:46
thanks so much, you explain it very well.
– FridayLi
Nov 20 '18 at 12:17
btw, 19700101001970.0, means the original date plus 19min and 70 seconds to me. and when I migrate the original data to mongo, and query it by aggregate, it seems like the origin query filter out user_download bigger than user created_at plus about 20mins.
– FridayLi
Nov 21 '18 at 1:55
add a comment |
Depending on the backend that you are using, you need your filter to generate sql like (for mysql):
...WHERE user_downloads.created_at >= DATE_ADD(users.created_at, INTERVAL 1 DAY)
SQLAlchemy doesn't convert arithmetic between a datetime
object and an InstrumentedAttribute
object into that DATE_ADD
(or equivalent depending on backend) function. So where you filter by this:
UserDownload.created_at >= User.created_at + timedelta(days=1)
it gets converted to this:
...WHERE user_downloads.created_at >= users.created_at + :created_at_1
where it treats timedelta(days=1)
as a literal value, and parameterises it. That is what the :created_at_1
is, a parameter that holds the place in the query for the timedelta
object which will be passed along with the query to the server (as a side note, in MySQL that timedelta
actually gets converted into a datetime
object that is epoch + 1 day, as MySQL doesn't have a native INTERVAL
type).
So to get the query to do what you want, you need to use the sqlalchemy.func
object to generate the server side function that you need. Continuing with the MySQL example, an appropriate query might be:
from sqlalchemy import func, text
q = session.query(UserDownload.uid).
join(User, UserDownload.uid == User.id).
filter(
UserDownload.created_at >=
func.DATE_ADD(
User.created_at,
text('INTERVAL 1 DAY')
)
)
Which generates:
SELECT user_downloads.uid
FROM user_downloads INNER JOIN users ON user_downloads.uid = users.id
WHERE user_downloads.created_at >= DATE_ADD(users.created_at, INTERVAL 1 DAY)
I found this question helpful: Using DATEADD in sqlalchemy
From the comments
since mysql can handle timedelta(days=1) params, why the query I used
fails.
OK, I'll try to go into more detail about your original query, but give me some latitude as I'm working this out as I go. Lets forget about the timedelta
for a second and just see what the sql generated is. So this:
session.query(UserDownload.uid).join(User, UserDownload.uid == User.id).filter(UserDownload.created_at >= User.created_at)
generates this sql:
SELECT user_downloads.uid AS user_downloads_uid
FROM user_downloads INNER JOIN users ON user_downloads.uid = users.id
WHERE user_downloads.created_at >= users.created_at
Nothing hard to grok there. Now when we add back in the timedelta
the sql generated is exactly the same except that we add a value on to users.created_at
which is represented by the bind parameter created_at_1
:
SELECT user_downloads.uid AS user_downloads_uid
FROM user_downloads INNER JOIN users ON user_downloads.uid = users.id
WHERE user_downloads.created_at >= users.created_at + %(created_at_1)s
So is the comparison executed first, or the addition? Running this query...
print(engine.execute(text("SELECT 3 >= 2 + 2")).fetchall())
... returns 0
(i.e. False
), which proves that the addition (2 + 2
) is resolved before the comparison (>=
). So safe to assume that happens in your query too and from that, the value of created_at_1
is added to users.created_at
prior to the comparison with user_downloads.created_at
. When I execute your query, this is the parameter value passed to the server:
{'created_at_1': datetime.datetime(1970, 1, 2, 0, 0)}
So even though you add timedelta(days=1)
to users.created_at
in your filter, SQLAlchemy actually passes a datetime
object equivalent to <epoch> + <timedelta>
, or in this case: datetime(1970, 1, 1, 0, 0) + timedelta(days=1)
or datetime(1970, 1, 2, 0, 0)
(which is the value that you see in the parameter value dictionary above).
So what exactly is the value of user.created_at + datetime(1970, 1, 2, 0, 0)
? I added a User
instance into the database with created_at = datetime(1970, 1, 1, 0, 0)
:
session.add(User(id=1, created_at=datetime(1970, 1, 1, 0, 0)))
session.commit()
Then ran this query:
engine.execute(text("SELECT created_at + :a FROM users"), a=datetime(1970, 1, 2, 0, 0)).fetchall()
which returned:
[(19700101001970.0,)]
That is the value of user.created_at
without any formatting, with the year portion of the datetime(1970, 1, 2, 0, 0)
concatenated to the end. So that is what your query is comparing to user_download.created_at
. For the sake of (relative) brevity, I'm not going to look into how the comparison of user_download.created_at
and that value works, but hopefully I've demonstrated that the end result of your query is not comparing user_download.created_at
with users.created_at
plus 1 day.
when I use query like: User.query.filter(User.created_at >
datetime.now() + timedelta(days=1)), it works fine.
Using that query, here is the generated sql:
SELECT users.id AS users_id, users.created_at AS users_created_at
FROM users
WHERE users.created_at > %(created_at_1)s
and the value passed on to the server is:
{'created_at_1': datetime.datetime(2018, 11, 21, 21, 38, 51, 670890)}
So you can see that the datetime + timedelta
part is resolved before being passed to the database and as such, the database operation is a simple comparison of a DATETIME
column to a datetime
value.
Thanks, It helps. But I still have a question, since mysql can handle timedelta(days=1) params, why the query I used fails. Btw, when I use query like: User.query.filter(User.created_at > datetime.now() + timedelta(days=1)), it works fine. Is it because you can't do the date add between python object and the column in database?
– FridayLi
Nov 20 '18 at 9:58
I've added to the answer above to try and address your extra questions.
– SuperShoot
Nov 20 '18 at 11:46
thanks so much, you explain it very well.
– FridayLi
Nov 20 '18 at 12:17
btw, 19700101001970.0, means the original date plus 19min and 70 seconds to me. and when I migrate the original data to mongo, and query it by aggregate, it seems like the origin query filter out user_download bigger than user created_at plus about 20mins.
– FridayLi
Nov 21 '18 at 1:55
add a comment |
Depending on the backend that you are using, you need your filter to generate sql like (for mysql):
...WHERE user_downloads.created_at >= DATE_ADD(users.created_at, INTERVAL 1 DAY)
SQLAlchemy doesn't convert arithmetic between a datetime
object and an InstrumentedAttribute
object into that DATE_ADD
(or equivalent depending on backend) function. So where you filter by this:
UserDownload.created_at >= User.created_at + timedelta(days=1)
it gets converted to this:
...WHERE user_downloads.created_at >= users.created_at + :created_at_1
where it treats timedelta(days=1)
as a literal value, and parameterises it. That is what the :created_at_1
is, a parameter that holds the place in the query for the timedelta
object which will be passed along with the query to the server (as a side note, in MySQL that timedelta
actually gets converted into a datetime
object that is epoch + 1 day, as MySQL doesn't have a native INTERVAL
type).
So to get the query to do what you want, you need to use the sqlalchemy.func
object to generate the server side function that you need. Continuing with the MySQL example, an appropriate query might be:
from sqlalchemy import func, text
q = session.query(UserDownload.uid).
join(User, UserDownload.uid == User.id).
filter(
UserDownload.created_at >=
func.DATE_ADD(
User.created_at,
text('INTERVAL 1 DAY')
)
)
Which generates:
SELECT user_downloads.uid
FROM user_downloads INNER JOIN users ON user_downloads.uid = users.id
WHERE user_downloads.created_at >= DATE_ADD(users.created_at, INTERVAL 1 DAY)
I found this question helpful: Using DATEADD in sqlalchemy
From the comments
since mysql can handle timedelta(days=1) params, why the query I used
fails.
OK, I'll try to go into more detail about your original query, but give me some latitude as I'm working this out as I go. Lets forget about the timedelta
for a second and just see what the sql generated is. So this:
session.query(UserDownload.uid).join(User, UserDownload.uid == User.id).filter(UserDownload.created_at >= User.created_at)
generates this sql:
SELECT user_downloads.uid AS user_downloads_uid
FROM user_downloads INNER JOIN users ON user_downloads.uid = users.id
WHERE user_downloads.created_at >= users.created_at
Nothing hard to grok there. Now when we add back in the timedelta
the sql generated is exactly the same except that we add a value on to users.created_at
which is represented by the bind parameter created_at_1
:
SELECT user_downloads.uid AS user_downloads_uid
FROM user_downloads INNER JOIN users ON user_downloads.uid = users.id
WHERE user_downloads.created_at >= users.created_at + %(created_at_1)s
So is the comparison executed first, or the addition? Running this query...
print(engine.execute(text("SELECT 3 >= 2 + 2")).fetchall())
... returns 0
(i.e. False
), which proves that the addition (2 + 2
) is resolved before the comparison (>=
). So safe to assume that happens in your query too and from that, the value of created_at_1
is added to users.created_at
prior to the comparison with user_downloads.created_at
. When I execute your query, this is the parameter value passed to the server:
{'created_at_1': datetime.datetime(1970, 1, 2, 0, 0)}
So even though you add timedelta(days=1)
to users.created_at
in your filter, SQLAlchemy actually passes a datetime
object equivalent to <epoch> + <timedelta>
, or in this case: datetime(1970, 1, 1, 0, 0) + timedelta(days=1)
or datetime(1970, 1, 2, 0, 0)
(which is the value that you see in the parameter value dictionary above).
So what exactly is the value of user.created_at + datetime(1970, 1, 2, 0, 0)
? I added a User
instance into the database with created_at = datetime(1970, 1, 1, 0, 0)
:
session.add(User(id=1, created_at=datetime(1970, 1, 1, 0, 0)))
session.commit()
Then ran this query:
engine.execute(text("SELECT created_at + :a FROM users"), a=datetime(1970, 1, 2, 0, 0)).fetchall()
which returned:
[(19700101001970.0,)]
That is the value of user.created_at
without any formatting, with the year portion of the datetime(1970, 1, 2, 0, 0)
concatenated to the end. So that is what your query is comparing to user_download.created_at
. For the sake of (relative) brevity, I'm not going to look into how the comparison of user_download.created_at
and that value works, but hopefully I've demonstrated that the end result of your query is not comparing user_download.created_at
with users.created_at
plus 1 day.
when I use query like: User.query.filter(User.created_at >
datetime.now() + timedelta(days=1)), it works fine.
Using that query, here is the generated sql:
SELECT users.id AS users_id, users.created_at AS users_created_at
FROM users
WHERE users.created_at > %(created_at_1)s
and the value passed on to the server is:
{'created_at_1': datetime.datetime(2018, 11, 21, 21, 38, 51, 670890)}
So you can see that the datetime + timedelta
part is resolved before being passed to the database and as such, the database operation is a simple comparison of a DATETIME
column to a datetime
value.
Depending on the backend that you are using, you need your filter to generate sql like (for mysql):
...WHERE user_downloads.created_at >= DATE_ADD(users.created_at, INTERVAL 1 DAY)
SQLAlchemy doesn't convert arithmetic between a datetime
object and an InstrumentedAttribute
object into that DATE_ADD
(or equivalent depending on backend) function. So where you filter by this:
UserDownload.created_at >= User.created_at + timedelta(days=1)
it gets converted to this:
...WHERE user_downloads.created_at >= users.created_at + :created_at_1
where it treats timedelta(days=1)
as a literal value, and parameterises it. That is what the :created_at_1
is, a parameter that holds the place in the query for the timedelta
object which will be passed along with the query to the server (as a side note, in MySQL that timedelta
actually gets converted into a datetime
object that is epoch + 1 day, as MySQL doesn't have a native INTERVAL
type).
So to get the query to do what you want, you need to use the sqlalchemy.func
object to generate the server side function that you need. Continuing with the MySQL example, an appropriate query might be:
from sqlalchemy import func, text
q = session.query(UserDownload.uid).
join(User, UserDownload.uid == User.id).
filter(
UserDownload.created_at >=
func.DATE_ADD(
User.created_at,
text('INTERVAL 1 DAY')
)
)
Which generates:
SELECT user_downloads.uid
FROM user_downloads INNER JOIN users ON user_downloads.uid = users.id
WHERE user_downloads.created_at >= DATE_ADD(users.created_at, INTERVAL 1 DAY)
I found this question helpful: Using DATEADD in sqlalchemy
From the comments
since mysql can handle timedelta(days=1) params, why the query I used
fails.
OK, I'll try to go into more detail about your original query, but give me some latitude as I'm working this out as I go. Lets forget about the timedelta
for a second and just see what the sql generated is. So this:
session.query(UserDownload.uid).join(User, UserDownload.uid == User.id).filter(UserDownload.created_at >= User.created_at)
generates this sql:
SELECT user_downloads.uid AS user_downloads_uid
FROM user_downloads INNER JOIN users ON user_downloads.uid = users.id
WHERE user_downloads.created_at >= users.created_at
Nothing hard to grok there. Now when we add back in the timedelta
the sql generated is exactly the same except that we add a value on to users.created_at
which is represented by the bind parameter created_at_1
:
SELECT user_downloads.uid AS user_downloads_uid
FROM user_downloads INNER JOIN users ON user_downloads.uid = users.id
WHERE user_downloads.created_at >= users.created_at + %(created_at_1)s
So is the comparison executed first, or the addition? Running this query...
print(engine.execute(text("SELECT 3 >= 2 + 2")).fetchall())
... returns 0
(i.e. False
), which proves that the addition (2 + 2
) is resolved before the comparison (>=
). So safe to assume that happens in your query too and from that, the value of created_at_1
is added to users.created_at
prior to the comparison with user_downloads.created_at
. When I execute your query, this is the parameter value passed to the server:
{'created_at_1': datetime.datetime(1970, 1, 2, 0, 0)}
So even though you add timedelta(days=1)
to users.created_at
in your filter, SQLAlchemy actually passes a datetime
object equivalent to <epoch> + <timedelta>
, or in this case: datetime(1970, 1, 1, 0, 0) + timedelta(days=1)
or datetime(1970, 1, 2, 0, 0)
(which is the value that you see in the parameter value dictionary above).
So what exactly is the value of user.created_at + datetime(1970, 1, 2, 0, 0)
? I added a User
instance into the database with created_at = datetime(1970, 1, 1, 0, 0)
:
session.add(User(id=1, created_at=datetime(1970, 1, 1, 0, 0)))
session.commit()
Then ran this query:
engine.execute(text("SELECT created_at + :a FROM users"), a=datetime(1970, 1, 2, 0, 0)).fetchall()
which returned:
[(19700101001970.0,)]
That is the value of user.created_at
without any formatting, with the year portion of the datetime(1970, 1, 2, 0, 0)
concatenated to the end. So that is what your query is comparing to user_download.created_at
. For the sake of (relative) brevity, I'm not going to look into how the comparison of user_download.created_at
and that value works, but hopefully I've demonstrated that the end result of your query is not comparing user_download.created_at
with users.created_at
plus 1 day.
when I use query like: User.query.filter(User.created_at >
datetime.now() + timedelta(days=1)), it works fine.
Using that query, here is the generated sql:
SELECT users.id AS users_id, users.created_at AS users_created_at
FROM users
WHERE users.created_at > %(created_at_1)s
and the value passed on to the server is:
{'created_at_1': datetime.datetime(2018, 11, 21, 21, 38, 51, 670890)}
So you can see that the datetime + timedelta
part is resolved before being passed to the database and as such, the database operation is a simple comparison of a DATETIME
column to a datetime
value.
edited Nov 20 '18 at 11:45
answered Nov 20 '18 at 9:50
SuperShootSuperShoot
1,845720
1,845720
Thanks, It helps. But I still have a question, since mysql can handle timedelta(days=1) params, why the query I used fails. Btw, when I use query like: User.query.filter(User.created_at > datetime.now() + timedelta(days=1)), it works fine. Is it because you can't do the date add between python object and the column in database?
– FridayLi
Nov 20 '18 at 9:58
I've added to the answer above to try and address your extra questions.
– SuperShoot
Nov 20 '18 at 11:46
thanks so much, you explain it very well.
– FridayLi
Nov 20 '18 at 12:17
btw, 19700101001970.0, means the original date plus 19min and 70 seconds to me. and when I migrate the original data to mongo, and query it by aggregate, it seems like the origin query filter out user_download bigger than user created_at plus about 20mins.
– FridayLi
Nov 21 '18 at 1:55
add a comment |
Thanks, It helps. But I still have a question, since mysql can handle timedelta(days=1) params, why the query I used fails. Btw, when I use query like: User.query.filter(User.created_at > datetime.now() + timedelta(days=1)), it works fine. Is it because you can't do the date add between python object and the column in database?
– FridayLi
Nov 20 '18 at 9:58
I've added to the answer above to try and address your extra questions.
– SuperShoot
Nov 20 '18 at 11:46
thanks so much, you explain it very well.
– FridayLi
Nov 20 '18 at 12:17
btw, 19700101001970.0, means the original date plus 19min and 70 seconds to me. and when I migrate the original data to mongo, and query it by aggregate, it seems like the origin query filter out user_download bigger than user created_at plus about 20mins.
– FridayLi
Nov 21 '18 at 1:55
Thanks, It helps. But I still have a question, since mysql can handle timedelta(days=1) params, why the query I used fails. Btw, when I use query like: User.query.filter(User.created_at > datetime.now() + timedelta(days=1)), it works fine. Is it because you can't do the date add between python object and the column in database?
– FridayLi
Nov 20 '18 at 9:58
Thanks, It helps. But I still have a question, since mysql can handle timedelta(days=1) params, why the query I used fails. Btw, when I use query like: User.query.filter(User.created_at > datetime.now() + timedelta(days=1)), it works fine. Is it because you can't do the date add between python object and the column in database?
– FridayLi
Nov 20 '18 at 9:58
I've added to the answer above to try and address your extra questions.
– SuperShoot
Nov 20 '18 at 11:46
I've added to the answer above to try and address your extra questions.
– SuperShoot
Nov 20 '18 at 11:46
thanks so much, you explain it very well.
– FridayLi
Nov 20 '18 at 12:17
thanks so much, you explain it very well.
– FridayLi
Nov 20 '18 at 12:17
btw, 19700101001970.0, means the original date plus 19min and 70 seconds to me. and when I migrate the original data to mongo, and query it by aggregate, it seems like the origin query filter out user_download bigger than user created_at plus about 20mins.
– FridayLi
Nov 21 '18 at 1:55
btw, 19700101001970.0, means the original date plus 19min and 70 seconds to me. and when I migrate the original data to mongo, and query it by aggregate, it seems like the origin query filter out user_download bigger than user created_at plus about 20mins.
– FridayLi
Nov 21 '18 at 1:55
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53388374%2fflask-sqlalchemy-compare-two-dates-of-two-related-tables%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
1
What database are you using?
– SuperShoot
Nov 20 '18 at 8:49
@SuperShoot I use Mysql as my database, and sqlalchemy as my orm in flask. SQLAlchemy==1.0.12
– FridayLi
Nov 20 '18 at 9:42