MongoDB 3.6.2 $out fails with duplicate key error
0
Here is the aggregation query which fails unexpectedly:
db.sq_lesson_user_lessons.aggregate([
{
"$match": {
lesson_id: {
"$in": [ObjectId("5bb6ec0a178353bbdecdd94d"), ObjectId("5bbf1e611783538013ce2f0a")]
},
status: { "$in": ['featured','started','pending','completed'] }
}
},
{
"$project": {
_id: 1,
user_profile_id: 1,
status: 1,
lesson_id: 1
}
},
{
"$out": "analytics_company_5bb6039598f17297c964fc54_sq_user_lessons"
}
])
assert: command failed: {
"operationTime" : Timestamp(1542715086, 67659),
"ok" : 0,
"errmsg" : "insert for $out failed: { lastOp: { ts: Timestamp(1542715086, 67657), t: 39 }, connectionId: 242551, err: "E11000 duplicate key error collection: api_smartquest_co_production.tmp.agg_out.637145 index: _id_ dup key: { : ObjectId('5bf22e554b8a982ada5e2828') }", code: 11000, codeName: "DuplicateKey", n: 0, ok: 1.0, operationTime: Timestamp(1542715086, 67657), $clusterTime: { clusterTime: Timestamp(1542715086, 67658), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } } }",
"code" : 16996,
"codeName" : "Location16996",
"$clusterTime" : {
"clusterTime" : Timestamp(1542715086, 67659),
"signature" : {
"hash" : BinData(0,"wvZz15/714/PHqAWywLpZlP4azQ="),
"keyId" : NumberLong("6606442824109916161")
}
}
} : aggregate failed
This aggregation results in about 300 thousand records. Sometimes the aggreagation works and sometimes it fails.
mongodb aggregation-framework
asked Nov 20 '18 at 9:15
prcoderprcoder
458
|
show 3 more comments
0
Here is the aggregation query which fails unexpectedly:
db.sq_lesson_user_lessons.aggregate([
{
"$match": {
lesson_id: {
"$in": [ObjectId("5bb6ec0a178353bbdecdd94d"), ObjectId("5bbf1e611783538013ce2f0a")]
},
status: { "$in": ['featured','started','pending','completed'] }
}
},
{
"$project": {
_id: 1,
user_profile_id: 1,
status: 1,
lesson_id: 1
}
},
{
"$out": "analytics_company_5bb6039598f17297c964fc54_sq_user_lessons"
}
])
assert: command failed: {
"operationTime" : Timestamp(1542715086, 67659),
"ok" : 0,
"errmsg" : "insert for $out failed: { lastOp: { ts: Timestamp(1542715086, 67657), t: 39 }, connectionId: 242551, err: "E11000 duplicate key error collection: api_smartquest_co_production.tmp.agg_out.637145 index: _id_ dup key: { : ObjectId('5bf22e554b8a982ada5e2828') }", code: 11000, codeName: "DuplicateKey", n: 0, ok: 1.0, operationTime: Timestamp(1542715086, 67657), $clusterTime: { clusterTime: Timestamp(1542715086, 67658), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } } }",
"code" : 16996,
"codeName" : "Location16996",
"$clusterTime" : {
"clusterTime" : Timestamp(1542715086, 67659),
"signature" : {
"hash" : BinData(0,"wvZz15/714/PHqAWywLpZlP4azQ="),
"keyId" : NumberLong("6606442824109916161")
}
}
} : aggregate failed
This aggregation results in about 300 thousand records. Sometimes the aggreagation works and sometimes it fails.
mongodb aggregation-framework
asked Nov 20 '18 at 9:15
prcoderprcoder
458
1
Are you certain it's this code where the error is throwing? I ask because there is nothing really being manipulated here at all, and such an error would typically come where you did an$unwindor similar stage within the pipeline. Also theto_ais pointless here since when a pipeline contains$outit returns no result.
– Neil Lunn
Nov 20 '18 at 9:21
Droptemp_collection, or at least remove indexes from there.
– Alex Blex
Nov 20 '18 at 9:23
1
@AlexBlex$outoverwrites ( read drops ) the collection anyway and the dup key is on_idin the error. At least that's the "current" behavior before 4.2, and the syntax for anything else is different anyway. So it's far more likely that this is not the code which is producing the error, just as stated earlier.
– Neil Lunn
Nov 20 '18 at 9:34
Hmm, I read docs.mongodb.com/manual/reference/operator/aggregation/out/… as it keeps the indexes. Agree on_idpoint tho.
– Alex Blex
Nov 20 '18 at 9:44
This is the exact aggregation which is throwing the error. To be precise about the query. Here are the match conditions I am using. { :$match => { :lesson_id => { :$in => ["id1","id2"] }, :status => { :$in => ['completed','started'] } } }
– prcoder
Nov 20 '18 at 11:05
|
show 3 more comments
0
0
0
Here is the aggregation query which fails unexpectedly:
db.sq_lesson_user_lessons.aggregate([
{
"$match": {
lesson_id: {
"$in": [ObjectId("5bb6ec0a178353bbdecdd94d"), ObjectId("5bbf1e611783538013ce2f0a")]
},
status: { "$in": ['featured','started','pending','completed'] }
}
},
{
"$project": {
_id: 1,
user_profile_id: 1,
status: 1,
lesson_id: 1
}
},
{
"$out": "analytics_company_5bb6039598f17297c964fc54_sq_user_lessons"
}
])
assert: command failed: {
"operationTime" : Timestamp(1542715086, 67659),
"ok" : 0,
"errmsg" : "insert for $out failed: { lastOp: { ts: Timestamp(1542715086, 67657), t: 39 }, connectionId: 242551, err: "E11000 duplicate key error collection: api_smartquest_co_production.tmp.agg_out.637145 index: _id_ dup key: { : ObjectId('5bf22e554b8a982ada5e2828') }", code: 11000, codeName: "DuplicateKey", n: 0, ok: 1.0, operationTime: Timestamp(1542715086, 67657), $clusterTime: { clusterTime: Timestamp(1542715086, 67658), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } } }",
"code" : 16996,
"codeName" : "Location16996",
"$clusterTime" : {
"clusterTime" : Timestamp(1542715086, 67659),
"signature" : {
"hash" : BinData(0,"wvZz15/714/PHqAWywLpZlP4azQ="),
"keyId" : NumberLong("6606442824109916161")
}
}
} : aggregate failed
This aggregation results in about 300 thousand records. Sometimes the aggreagation works and sometimes it fails.
mongodb aggregation-framework
asked Nov 20 '18 at 9:15
prcoderprcoder
458
Here is the aggregation query which fails unexpectedly:
db.sq_lesson_user_lessons.aggregate([
{
"$match": {
lesson_id: {
"$in": [ObjectId("5bb6ec0a178353bbdecdd94d"), ObjectId("5bbf1e611783538013ce2f0a")]
},
status: { "$in": ['featured','started','pending','completed'] }
}
},
{
"$project": {
_id: 1,
user_profile_id: 1,
status: 1,
lesson_id: 1
}
},
{
"$out": "analytics_company_5bb6039598f17297c964fc54_sq_user_lessons"
}
])
assert: command failed: {
"operationTime" : Timestamp(1542715086, 67659),
"ok" : 0,
"errmsg" : "insert for $out failed: { lastOp: { ts: Timestamp(1542715086, 67657), t: 39 }, connectionId: 242551, err: "E11000 duplicate key error collection: api_smartquest_co_production.tmp.agg_out.637145 index: _id_ dup key: { : ObjectId('5bf22e554b8a982ada5e2828') }", code: 11000, codeName: "DuplicateKey", n: 0, ok: 1.0, operationTime: Timestamp(1542715086, 67657), $clusterTime: { clusterTime: Timestamp(1542715086, 67658), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } } }",
"code" : 16996,
"codeName" : "Location16996",
"$clusterTime" : {
"clusterTime" : Timestamp(1542715086, 67659),
"signature" : {
"hash" : BinData(0,"wvZz15/714/PHqAWywLpZlP4azQ="),
"keyId" : NumberLong("6606442824109916161")
}
}
} : aggregate failed
This aggregation results in about 300 thousand records. Sometimes the aggreagation works and sometimes it fails.
mongodb aggregation-framework
mongodb aggregation-framework
asked Nov 20 '18 at 9:15
prcoderprcoder
458
asked Nov 20 '18 at 9:15
prcoderprcoder
458
edited Nov 20 '18 at 12:03
prcoder
asked Nov 20 '18 at 9:15
prcoderprcoder
458
asked Nov 20 '18 at 9:15
prcoderprcoder
458
asked Nov 20 '18 at 9:15
prcoderprcoder
458
458
1
Are you certain it's this code where the error is throwing? I ask because there is nothing really being manipulated here at all, and such an error would typically come where you did an$unwindor similar stage within the pipeline. Also theto_ais pointless here since when a pipeline contains$outit returns no result.
– Neil Lunn
Nov 20 '18 at 9:21
Droptemp_collection, or at least remove indexes from there.
– Alex Blex
Nov 20 '18 at 9:23
1
@AlexBlex$outoverwrites ( read drops ) the collection anyway and the dup key is on_idin the error. At least that's the "current" behavior before 4.2, and the syntax for anything else is different anyway. So it's far more likely that this is not the code which is producing the error, just as stated earlier.
– Neil Lunn
Nov 20 '18 at 9:34
Hmm, I read docs.mongodb.com/manual/reference/operator/aggregation/out/… as it keeps the indexes. Agree on_idpoint tho.
– Alex Blex
Nov 20 '18 at 9:44
This is the exact aggregation which is throwing the error. To be precise about the query. Here are the match conditions I am using. { :$match => { :lesson_id => { :$in => ["id1","id2"] }, :status => { :$in => ['completed','started'] } } }
– prcoder
Nov 20 '18 at 11:05
|
show 3 more comments
1
Are you certain it's this code where the error is throwing? I ask because there is nothing really being manipulated here at all, and such an error would typically come where you did an$unwindor similar stage within the pipeline. Also theto_ais pointless here since when a pipeline contains$outit returns no result.
– Neil Lunn
Nov 20 '18 at 9:21
Droptemp_collection, or at least remove indexes from there.
– Alex Blex
Nov 20 '18 at 9:23
1
@AlexBlex$outoverwrites ( read drops ) the collection anyway and the dup key is on_idin the error. At least that's the "current" behavior before 4.2, and the syntax for anything else is different anyway. So it's far more likely that this is not the code which is producing the error, just as stated earlier.
– Neil Lunn
Nov 20 '18 at 9:34
Hmm, I read docs.mongodb.com/manual/reference/operator/aggregation/out/… as it keeps the indexes. Agree on_idpoint tho.
– Alex Blex
Nov 20 '18 at 9:44
This is the exact aggregation which is throwing the error. To be precise about the query. Here are the match conditions I am using. { :$match => { :lesson_id => { :$in => ["id1","id2"] }, :status => { :$in => ['completed','started'] } } }
– prcoder
Nov 20 '18 at 11:05
1
1
Are you certain it's this code where the error is throwing? I ask because there is nothing really being manipulated here at all, and such an error would typically come where you did an
$unwind or similar stage within the pipeline. Also the to_a is pointless here since when a pipeline contains $out it returns no result.– Neil Lunn
Nov 20 '18 at 9:21
Are you certain it's this code where the error is throwing? I ask because there is nothing really being manipulated here at all, and such an error would typically come where you did an
$unwind or similar stage within the pipeline. Also the to_a is pointless here since when a pipeline contains $out it returns no result.– Neil Lunn
Nov 20 '18 at 9:21
Drop
temp_collection, or at least remove indexes from there.– Alex Blex
Nov 20 '18 at 9:23
Drop
temp_collection, or at least remove indexes from there.– Alex Blex
Nov 20 '18 at 9:23
1
1
@AlexBlex
$out overwrites ( read drops ) the collection anyway and the dup key is on _id in the error. At least that's the "current" behavior before 4.2, and the syntax for anything else is different anyway. So it's far more likely that this is not the code which is producing the error, just as stated earlier.– Neil Lunn
Nov 20 '18 at 9:34
@AlexBlex
$out overwrites ( read drops ) the collection anyway and the dup key is on _id in the error. At least that's the "current" behavior before 4.2, and the syntax for anything else is different anyway. So it's far more likely that this is not the code which is producing the error, just as stated earlier.– Neil Lunn
Nov 20 '18 at 9:34
Hmm, I read docs.mongodb.com/manual/reference/operator/aggregation/out/… as it keeps the indexes. Agree on
_id point tho.– Alex Blex
Nov 20 '18 at 9:44
Hmm, I read docs.mongodb.com/manual/reference/operator/aggregation/out/… as it keeps the indexes. Agree on
_id point tho.– Alex Blex
Nov 20 '18 at 9:44
This is the exact aggregation which is throwing the error. To be precise about the query. Here are the match conditions I am using. { :$match => { :lesson_id => { :$in => ["id1","id2"] }, :status => { :$in => ['completed','started'] } } }
– prcoder
Nov 20 '18 at 11:05
This is the exact aggregation which is throwing the error. To be precise about the query. Here are the match conditions I am using. { :$match => { :lesson_id => { :$in => ["id1","id2"] }, :status => { :$in => ['completed','started'] } } }
– prcoder
Nov 20 '18 at 11:05
|
show 3 more comments
1 Answer
1
active
oldest
votes
0
This was answered by Daniel Hatcher from MongoDB with the below comment:
https://jira.mongodb.org/browse/SERVER-38212?focusedCommentId=2067603&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-2067603
CRUX of the answer is:
As the aggregation is searching through the large collection to return results, it is possible that some documents are being returned multiple times. This is related to one of the concepts within MongoDB's read isolation.
answered Dec 10 '18 at 10:41
prcoderprcoder
458
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
draft saved
draft discarded
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53389686%2fmongodb-3-6-2-out-fails-with-duplicate-key-error%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
0
This was answered by Daniel Hatcher from MongoDB with the below comment:
https://jira.mongodb.org/browse/SERVER-38212?focusedCommentId=2067603&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-2067603
CRUX of the answer is:
As the aggregation is searching through the large collection to return results, it is possible that some documents are being returned multiple times. This is related to one of the concepts within MongoDB's read isolation.
answered Dec 10 '18 at 10:41
prcoderprcoder
458
add a comment |
0
This was answered by Daniel Hatcher from MongoDB with the below comment:
https://jira.mongodb.org/browse/SERVER-38212?focusedCommentId=2067603&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-2067603
CRUX of the answer is:
As the aggregation is searching through the large collection to return results, it is possible that some documents are being returned multiple times. This is related to one of the concepts within MongoDB's read isolation.
answered Dec 10 '18 at 10:41
prcoderprcoder
458
add a comment |
0
0
0
This was answered by Daniel Hatcher from MongoDB with the below comment:
https://jira.mongodb.org/browse/SERVER-38212?focusedCommentId=2067603&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-2067603
CRUX of the answer is:
As the aggregation is searching through the large collection to return results, it is possible that some documents are being returned multiple times. This is related to one of the concepts within MongoDB's read isolation.
answered Dec 10 '18 at 10:41
prcoderprcoder
458
This was answered by Daniel Hatcher from MongoDB with the below comment:
https://jira.mongodb.org/browse/SERVER-38212?focusedCommentId=2067603&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-2067603
CRUX of the answer is:
As the aggregation is searching through the large collection to return results, it is possible that some documents are being returned multiple times. This is related to one of the concepts within MongoDB's read isolation.
answered Dec 10 '18 at 10:41
prcoderprcoder
458
answered Dec 10 '18 at 10:41
prcoderprcoder
458
answered Dec 10 '18 at 10:41
prcoderprcoder
458
answered Dec 10 '18 at 10:41
prcoderprcoder
458
458
add a comment |
add a comment |
draft saved
draft discarded
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
draft saved
draft discarded
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53389686%2fmongodb-3-6-2-out-fails-with-duplicate-key-error%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
1
Are you certain it's this code where the error is throwing? I ask because there is nothing really being manipulated here at all, and such an error would typically come where you did an
$unwindor similar stage within the pipeline. Also theto_ais pointless here since when a pipeline contains$outit returns no result.– Neil Lunn
Nov 20 '18 at 9:21
Drop
temp_collection, or at least remove indexes from there.– Alex Blex
Nov 20 '18 at 9:23
1
@AlexBlex
$outoverwrites ( read drops ) the collection anyway and the dup key is on_idin the error. At least that's the "current" behavior before 4.2, and the syntax for anything else is different anyway. So it's far more likely that this is not the code which is producing the error, just as stated earlier.– Neil Lunn
Nov 20 '18 at 9:34
Hmm, I read docs.mongodb.com/manual/reference/operator/aggregation/out/… as it keeps the indexes. Agree on
_idpoint tho.– Alex Blex
Nov 20 '18 at 9:44
This is the exact aggregation which is throwing the error. To be precise about the query. Here are the match conditions I am using. { :$match => { :lesson_id => { :$in => ["id1","id2"] }, :status => { :$in => ['completed','started'] } } }
– prcoder
Nov 20 '18 at 11:05