MongoDB Cursor Snapshot with C# Driver
2
I have a large collection that I need to iterate through. I am using a cursor for this.
However, I noticed I am getting duplicate records sometimes from this cursor. I did some reading and found that I can address this by using a snapshot.
Cursor Isolation
As a cursor returns documents, other operations may interleave with the query. For the MMAPv1 storage engine, intervening write operations on a document may result in a cursor that returns a document more than once if that document has changed. To handle this situation, see the information on Cursor Snapshot.
However, I am unable to find this method in the C# Driver (Version 2.7). I don't see it on FindFluent or in FindOptions.
Can someone point me to an example of how I can create a cursor snapshot via the C# MongoDB Driver?
Here is what my code looks like.
private async Task<int> IterateViaCursor<P>(FilterDefinition<User> filter, ProjectionDefinition<User, P> pdef, Func<IEnumerable<P>, Task> callback, int batchSize)
where P : class
{
var o = new FindOptions { BatchSize = batchSize };
var count = 0;
var findFluent = Users.Find(filter, o).Project(pdef);
using (var cursor = await findFluent.ToCursorAsync())
{
while (await cursor.MoveNextAsync())
{
var batchCount = cursor.Current.Count();
count += batchCount;
if (batchCount > 0)
await callback(cursor.Current.ToList());
}
}
return count;
}
c# mongodb cursor
asked Nov 20 '18 at 1:59
Vadim RybakVadim Rybak
1,1971525
|
show 1 more comment
2
I have a large collection that I need to iterate through. I am using a cursor for this.
However, I noticed I am getting duplicate records sometimes from this cursor. I did some reading and found that I can address this by using a snapshot.
Cursor Isolation
As a cursor returns documents, other operations may interleave with the query. For the MMAPv1 storage engine, intervening write operations on a document may result in a cursor that returns a document more than once if that document has changed. To handle this situation, see the information on Cursor Snapshot.
However, I am unable to find this method in the C# Driver (Version 2.7). I don't see it on FindFluent or in FindOptions.
Can someone point me to an example of how I can create a cursor snapshot via the C# MongoDB Driver?
Here is what my code looks like.
private async Task<int> IterateViaCursor<P>(FilterDefinition<User> filter, ProjectionDefinition<User, P> pdef, Func<IEnumerable<P>, Task> callback, int batchSize)
where P : class
{
var o = new FindOptions { BatchSize = batchSize };
var count = 0;
var findFluent = Users.Find(filter, o).Project(pdef);
using (var cursor = await findFluent.ToCursorAsync())
{
while (await cursor.MoveNextAsync())
{
var batchCount = cursor.Current.Count();
count += batchCount;
if (batchCount > 0)
await callback(cursor.Current.ToList());
}
}
return count;
}
c# mongodb cursor
asked Nov 20 '18 at 1:59
Vadim RybakVadim Rybak
1,1971525
1
Snapshot was deprecated and then removed for MongoDB 4.0. jira.mongodb.org/browse/SERVER-32173. So you 'read some old posts" which talked about snapshot, but this was really just a "band aid" and not a real solution, hence the removal. Instead, explain to us "why" you think you need to iterate a long cursor and the actual code you are using showing you these "duplicates". Even if you are using an older MongoDB server, the solution to your code should not be one using the snapshot option.
– Neil Lunn
Nov 20 '18 at 2:05
The reason I want to use a snapshot is because I am getting duplicate records when iterating my cursor. I updated my question with some more information
– Vadim Rybak
Nov 20 '18 at 2:59
Can you provide an MCVE that actually reproduces the behavior then? The code provided seems fine, so any issue is likely more related to something else that is happening or other configuration. Note that if you follow though the links from the issue raised to deprecate and remove, it should become apparent that "snapshots" never actually did what most people were expecting them to do, and are also tied to the now removed MMAPv1 storage engine. Newer storage engines have different read isolation by default.
– Neil Lunn
Nov 20 '18 at 3:08
Was about to talk about something else, but is this the actual code in use? If so then all you are really doing here is "counting" results. Unless there is some other purpose for the code not included here, then there is a far more effective way to to that, instead of iterating a cursor and incrementing a counter.
– Neil Lunn
Nov 20 '18 at 3:13
SpecificallyMongoCollection.Count(IMongoQuery), which I just had to check because some language API's now call thiscountDocuments()or similar, but it's not changed here. This is the recommended reliable way to count query results.
– Neil Lunn
Nov 20 '18 at 3:18
|
show 1 more comment
2
2
2
I have a large collection that I need to iterate through. I am using a cursor for this.
However, I noticed I am getting duplicate records sometimes from this cursor. I did some reading and found that I can address this by using a snapshot.
Cursor Isolation
As a cursor returns documents, other operations may interleave with the query. For the MMAPv1 storage engine, intervening write operations on a document may result in a cursor that returns a document more than once if that document has changed. To handle this situation, see the information on Cursor Snapshot.
However, I am unable to find this method in the C# Driver (Version 2.7). I don't see it on FindFluent or in FindOptions.
Can someone point me to an example of how I can create a cursor snapshot via the C# MongoDB Driver?
Here is what my code looks like.
private async Task<int> IterateViaCursor<P>(FilterDefinition<User> filter, ProjectionDefinition<User, P> pdef, Func<IEnumerable<P>, Task> callback, int batchSize)
where P : class
{
var o = new FindOptions { BatchSize = batchSize };
var count = 0;
var findFluent = Users.Find(filter, o).Project(pdef);
using (var cursor = await findFluent.ToCursorAsync())
{
while (await cursor.MoveNextAsync())
{
var batchCount = cursor.Current.Count();
count += batchCount;
if (batchCount > 0)
await callback(cursor.Current.ToList());
}
}
return count;
}
c# mongodb cursor
asked Nov 20 '18 at 1:59
Vadim RybakVadim Rybak
1,1971525
I have a large collection that I need to iterate through. I am using a cursor for this.
However, I noticed I am getting duplicate records sometimes from this cursor. I did some reading and found that I can address this by using a snapshot.
Cursor Isolation
As a cursor returns documents, other operations may interleave with the query. For the MMAPv1 storage engine, intervening write operations on a document may result in a cursor that returns a document more than once if that document has changed. To handle this situation, see the information on Cursor Snapshot.
However, I am unable to find this method in the C# Driver (Version 2.7). I don't see it on FindFluent or in FindOptions.
Can someone point me to an example of how I can create a cursor snapshot via the C# MongoDB Driver?
Here is what my code looks like.
private async Task<int> IterateViaCursor<P>(FilterDefinition<User> filter, ProjectionDefinition<User, P> pdef, Func<IEnumerable<P>, Task> callback, int batchSize)
where P : class
{
var o = new FindOptions { BatchSize = batchSize };
var count = 0;
var findFluent = Users.Find(filter, o).Project(pdef);
using (var cursor = await findFluent.ToCursorAsync())
{
while (await cursor.MoveNextAsync())
{
var batchCount = cursor.Current.Count();
count += batchCount;
if (batchCount > 0)
await callback(cursor.Current.ToList());
}
}
return count;
}
c# mongodb cursor
c# mongodb cursor
asked Nov 20 '18 at 1:59
Vadim RybakVadim Rybak
1,1971525
asked Nov 20 '18 at 1:59
Vadim RybakVadim Rybak
1,1971525
edited Nov 20 '18 at 2:58
Vadim Rybak
asked Nov 20 '18 at 1:59
Vadim RybakVadim Rybak
1,1971525
asked Nov 20 '18 at 1:59
Vadim RybakVadim Rybak
1,1971525
asked Nov 20 '18 at 1:59
Vadim RybakVadim Rybak
1,1971525
1,1971525
1
Snapshot was deprecated and then removed for MongoDB 4.0. jira.mongodb.org/browse/SERVER-32173. So you 'read some old posts" which talked about snapshot, but this was really just a "band aid" and not a real solution, hence the removal. Instead, explain to us "why" you think you need to iterate a long cursor and the actual code you are using showing you these "duplicates". Even if you are using an older MongoDB server, the solution to your code should not be one using the snapshot option.
– Neil Lunn
Nov 20 '18 at 2:05
The reason I want to use a snapshot is because I am getting duplicate records when iterating my cursor. I updated my question with some more information
– Vadim Rybak
Nov 20 '18 at 2:59
Can you provide an MCVE that actually reproduces the behavior then? The code provided seems fine, so any issue is likely more related to something else that is happening or other configuration. Note that if you follow though the links from the issue raised to deprecate and remove, it should become apparent that "snapshots" never actually did what most people were expecting them to do, and are also tied to the now removed MMAPv1 storage engine. Newer storage engines have different read isolation by default.
– Neil Lunn
Nov 20 '18 at 3:08
Was about to talk about something else, but is this the actual code in use? If so then all you are really doing here is "counting" results. Unless there is some other purpose for the code not included here, then there is a far more effective way to to that, instead of iterating a cursor and incrementing a counter.
– Neil Lunn
Nov 20 '18 at 3:13
SpecificallyMongoCollection.Count(IMongoQuery), which I just had to check because some language API's now call thiscountDocuments()or similar, but it's not changed here. This is the recommended reliable way to count query results.
– Neil Lunn
Nov 20 '18 at 3:18
|
show 1 more comment
1
Snapshot was deprecated and then removed for MongoDB 4.0. jira.mongodb.org/browse/SERVER-32173. So you 'read some old posts" which talked about snapshot, but this was really just a "band aid" and not a real solution, hence the removal. Instead, explain to us "why" you think you need to iterate a long cursor and the actual code you are using showing you these "duplicates". Even if you are using an older MongoDB server, the solution to your code should not be one using the snapshot option.
– Neil Lunn
Nov 20 '18 at 2:05
The reason I want to use a snapshot is because I am getting duplicate records when iterating my cursor. I updated my question with some more information
– Vadim Rybak
Nov 20 '18 at 2:59
Can you provide an MCVE that actually reproduces the behavior then? The code provided seems fine, so any issue is likely more related to something else that is happening or other configuration. Note that if you follow though the links from the issue raised to deprecate and remove, it should become apparent that "snapshots" never actually did what most people were expecting them to do, and are also tied to the now removed MMAPv1 storage engine. Newer storage engines have different read isolation by default.
– Neil Lunn
Nov 20 '18 at 3:08
Was about to talk about something else, but is this the actual code in use? If so then all you are really doing here is "counting" results. Unless there is some other purpose for the code not included here, then there is a far more effective way to to that, instead of iterating a cursor and incrementing a counter.
– Neil Lunn
Nov 20 '18 at 3:13
SpecificallyMongoCollection.Count(IMongoQuery), which I just had to check because some language API's now call thiscountDocuments()or similar, but it's not changed here. This is the recommended reliable way to count query results.
– Neil Lunn
Nov 20 '18 at 3:18
1
1
Snapshot was deprecated and then removed for MongoDB 4.0. jira.mongodb.org/browse/SERVER-32173. So you 'read some old posts" which talked about snapshot, but this was really just a "band aid" and not a real solution, hence the removal. Instead, explain to us "why" you think you need to iterate a long cursor and the actual code you are using showing you these "duplicates". Even if you are using an older MongoDB server, the solution to your code should not be one using the snapshot option.
– Neil Lunn
Nov 20 '18 at 2:05
Snapshot was deprecated and then removed for MongoDB 4.0. jira.mongodb.org/browse/SERVER-32173. So you 'read some old posts" which talked about snapshot, but this was really just a "band aid" and not a real solution, hence the removal. Instead, explain to us "why" you think you need to iterate a long cursor and the actual code you are using showing you these "duplicates". Even if you are using an older MongoDB server, the solution to your code should not be one using the snapshot option.
– Neil Lunn
Nov 20 '18 at 2:05
The reason I want to use a snapshot is because I am getting duplicate records when iterating my cursor. I updated my question with some more information
– Vadim Rybak
Nov 20 '18 at 2:59
The reason I want to use a snapshot is because I am getting duplicate records when iterating my cursor. I updated my question with some more information
– Vadim Rybak
Nov 20 '18 at 2:59
Can you provide an MCVE that actually reproduces the behavior then? The code provided seems fine, so any issue is likely more related to something else that is happening or other configuration. Note that if you follow though the links from the issue raised to deprecate and remove, it should become apparent that "snapshots" never actually did what most people were expecting them to do, and are also tied to the now removed MMAPv1 storage engine. Newer storage engines have different read isolation by default.
– Neil Lunn
Nov 20 '18 at 3:08
Can you provide an MCVE that actually reproduces the behavior then? The code provided seems fine, so any issue is likely more related to something else that is happening or other configuration. Note that if you follow though the links from the issue raised to deprecate and remove, it should become apparent that "snapshots" never actually did what most people were expecting them to do, and are also tied to the now removed MMAPv1 storage engine. Newer storage engines have different read isolation by default.
– Neil Lunn
Nov 20 '18 at 3:08
Was about to talk about something else, but is this the actual code in use? If so then all you are really doing here is "counting" results. Unless there is some other purpose for the code not included here, then there is a far more effective way to to that, instead of iterating a cursor and incrementing a counter.
– Neil Lunn
Nov 20 '18 at 3:13
Was about to talk about something else, but is this the actual code in use? If so then all you are really doing here is "counting" results. Unless there is some other purpose for the code not included here, then there is a far more effective way to to that, instead of iterating a cursor and incrementing a counter.
– Neil Lunn
Nov 20 '18 at 3:13
Specifically
MongoCollection.Count(IMongoQuery), which I just had to check because some language API's now call this countDocuments() or similar, but it's not changed here. This is the recommended reliable way to count query results.– Neil Lunn
Nov 20 '18 at 3:18
Specifically
MongoCollection.Count(IMongoQuery), which I just had to check because some language API's now call this countDocuments() or similar, but it's not changed here. This is the recommended reliable way to count query results.– Neil Lunn
Nov 20 '18 at 3:18
|
show 1 more comment
0
active
oldest
votes
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
draft saved
draft discarded
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53385168%2fmongodb-cursor-snapshot-with-c-sharp-driver%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
0
active
oldest
votes
0
active
oldest
votes
active
oldest
votes
active
oldest
votes
draft saved
draft discarded
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
draft saved
draft discarded
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53385168%2fmongodb-cursor-snapshot-with-c-sharp-driver%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
1
Snapshot was deprecated and then removed for MongoDB 4.0. jira.mongodb.org/browse/SERVER-32173. So you 'read some old posts" which talked about snapshot, but this was really just a "band aid" and not a real solution, hence the removal. Instead, explain to us "why" you think you need to iterate a long cursor and the actual code you are using showing you these "duplicates". Even if you are using an older MongoDB server, the solution to your code should not be one using the snapshot option.
– Neil Lunn
Nov 20 '18 at 2:05
The reason I want to use a snapshot is because I am getting duplicate records when iterating my cursor. I updated my question with some more information
– Vadim Rybak
Nov 20 '18 at 2:59
Can you provide an MCVE that actually reproduces the behavior then? The code provided seems fine, so any issue is likely more related to something else that is happening or other configuration. Note that if you follow though the links from the issue raised to deprecate and remove, it should become apparent that "snapshots" never actually did what most people were expecting them to do, and are also tied to the now removed MMAPv1 storage engine. Newer storage engines have different read isolation by default.
– Neil Lunn
Nov 20 '18 at 3:08
Was about to talk about something else, but is this the actual code in use? If so then all you are really doing here is "counting" results. Unless there is some other purpose for the code not included here, then there is a far more effective way to to that, instead of iterating a cursor and incrementing a counter.
– Neil Lunn
Nov 20 '18 at 3:13
Specifically
MongoCollection.Count(IMongoQuery), which I just had to check because some language API's now call thiscountDocuments()or similar, but it's not changed here. This is the recommended reliable way to count query results.– Neil Lunn
Nov 20 '18 at 3:18