Error handling while using chain of groups in celery
I have 100 tasks. But I want to process only 4 tasks at a time. Once these 4 tasks get completed i want to run the next set of 4 tasks.
This can be done by grouping tasks in sets of 4 and then chaining them.
But some tasks may fail in between, how can I handle these errors and take appropriate actions, so that I retry for only the tasks that failed.
As per my knowledge in a chain if one task fails, the subsequent tasks in the chain will not run. So if I implement chain of groups, if any tasks in a group fails, the entire chain will fail.
Suggest me a proper error handling method for this, also any better idea for implementing this will be appreciated.
I have disabled the result backend.
A little background - I have a cron running every 30s implemented using celery-beat which picks up 100 new tasks at a time. So all these chaining and grouping has to be done inside the cron function.
I am using celery 3.1.25
django celery celery-task celerybeat djcelery
asked Nov 17 '18 at 14:26
DUDE_MXPDUDE_MXP
46449
add a comment |
I have 100 tasks. But I want to process only 4 tasks at a time. Once these 4 tasks get completed i want to run the next set of 4 tasks.
This can be done by grouping tasks in sets of 4 and then chaining them.
But some tasks may fail in between, how can I handle these errors and take appropriate actions, so that I retry for only the tasks that failed.
As per my knowledge in a chain if one task fails, the subsequent tasks in the chain will not run. So if I implement chain of groups, if any tasks in a group fails, the entire chain will fail.
Suggest me a proper error handling method for this, also any better idea for implementing this will be appreciated.
I have disabled the result backend.
A little background - I have a cron running every 30s implemented using celery-beat which picks up 100 new tasks at a time. So all these chaining and grouping has to be done inside the cron function.
I am using celery 3.1.25
django celery celery-task celerybeat djcelery
asked Nov 17 '18 at 14:26
DUDE_MXPDUDE_MXP
46449
Do the next set of tasks require the first set of tasks to be run?
– schillingt
Nov 17 '18 at 15:53
@schillingt no it's not required. All tasks are independent of each other.
– DUDE_MXP
Nov 17 '18 at 16:00
add a comment |
I have 100 tasks. But I want to process only 4 tasks at a time. Once these 4 tasks get completed i want to run the next set of 4 tasks.
This can be done by grouping tasks in sets of 4 and then chaining them.
But some tasks may fail in between, how can I handle these errors and take appropriate actions, so that I retry for only the tasks that failed.
As per my knowledge in a chain if one task fails, the subsequent tasks in the chain will not run. So if I implement chain of groups, if any tasks in a group fails, the entire chain will fail.
Suggest me a proper error handling method for this, also any better idea for implementing this will be appreciated.
I have disabled the result backend.
A little background - I have a cron running every 30s implemented using celery-beat which picks up 100 new tasks at a time. So all these chaining and grouping has to be done inside the cron function.
I am using celery 3.1.25
django celery celery-task celerybeat djcelery
asked Nov 17 '18 at 14:26
DUDE_MXPDUDE_MXP
46449
I have 100 tasks. But I want to process only 4 tasks at a time. Once these 4 tasks get completed i want to run the next set of 4 tasks.
This can be done by grouping tasks in sets of 4 and then chaining them.
But some tasks may fail in between, how can I handle these errors and take appropriate actions, so that I retry for only the tasks that failed.
As per my knowledge in a chain if one task fails, the subsequent tasks in the chain will not run. So if I implement chain of groups, if any tasks in a group fails, the entire chain will fail.
Suggest me a proper error handling method for this, also any better idea for implementing this will be appreciated.
I have disabled the result backend.
A little background - I have a cron running every 30s implemented using celery-beat which picks up 100 new tasks at a time. So all these chaining and grouping has to be done inside the cron function.
I am using celery 3.1.25
django celery celery-task celerybeat djcelery
django celery celery-task celerybeat djcelery
asked Nov 17 '18 at 14:26
DUDE_MXPDUDE_MXP
46449
asked Nov 17 '18 at 14:26
DUDE_MXPDUDE_MXP
46449
edited Nov 17 '18 at 14:40
DUDE_MXP
asked Nov 17 '18 at 14:26
DUDE_MXPDUDE_MXP
46449
asked Nov 17 '18 at 14:26
DUDE_MXPDUDE_MXP
46449
asked Nov 17 '18 at 14:26
DUDE_MXPDUDE_MXP
46449
46449
Do the next set of tasks require the first set of tasks to be run?
– schillingt
Nov 17 '18 at 15:53
@schillingt no it's not required. All tasks are independent of each other.
– DUDE_MXP
Nov 17 '18 at 16:00
add a comment |
Do the next set of tasks require the first set of tasks to be run?
– schillingt
Nov 17 '18 at 15:53
@schillingt no it's not required. All tasks are independent of each other.
– DUDE_MXP
Nov 17 '18 at 16:00
Do the next set of tasks require the first set of tasks to be run?
– schillingt
Nov 17 '18 at 15:53
Do the next set of tasks require the first set of tasks to be run?
– schillingt
Nov 17 '18 at 15:53
@schillingt no it's not required. All tasks are independent of each other.
– DUDE_MXP
Nov 17 '18 at 16:00
@schillingt no it's not required. All tasks are independent of each other.
– DUDE_MXP
Nov 17 '18 at 16:00
add a comment |
1 Answer
1
active
oldest
votes
Don't chain the tasks together if they are independent. Adjust the settings on the celery worker for a specific queue to limit the number of workers to 4 using --concurrency 4 or -c 4.
answered Nov 17 '18 at 16:03
schillingtschillingt
5,30211722
you mean that I feed all the 100 tasks to the queue, but since no of worker is 4 only 4 tasks will be processed at a time
– DUDE_MXP
Nov 17 '18 at 16:07
Basically, yes. You might have to tune some other settings, but that is the approach I would take.
– schillingt
Nov 17 '18 at 16:08
Right now i'm running only 1 worker with concurrency of 4, I think this configuration alone can help me in achieving what i want. I just need to feel all of them at once to the queue.
– DUDE_MXP
Nov 17 '18 at 16:14
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53352122%2ferror-handling-while-using-chain-of-groups-in-celery%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
Don't chain the tasks together if they are independent. Adjust the settings on the celery worker for a specific queue to limit the number of workers to 4 using --concurrency 4 or -c 4.
answered Nov 17 '18 at 16:03
schillingtschillingt
5,30211722
you mean that I feed all the 100 tasks to the queue, but since no of worker is 4 only 4 tasks will be processed at a time
– DUDE_MXP
Nov 17 '18 at 16:07
Basically, yes. You might have to tune some other settings, but that is the approach I would take.
– schillingt
Nov 17 '18 at 16:08
Right now i'm running only 1 worker with concurrency of 4, I think this configuration alone can help me in achieving what i want. I just need to feel all of them at once to the queue.
– DUDE_MXP
Nov 17 '18 at 16:14
add a comment |
Don't chain the tasks together if they are independent. Adjust the settings on the celery worker for a specific queue to limit the number of workers to 4 using --concurrency 4 or -c 4.
answered Nov 17 '18 at 16:03
schillingtschillingt
5,30211722
you mean that I feed all the 100 tasks to the queue, but since no of worker is 4 only 4 tasks will be processed at a time
– DUDE_MXP
Nov 17 '18 at 16:07
Basically, yes. You might have to tune some other settings, but that is the approach I would take.
– schillingt
Nov 17 '18 at 16:08
Right now i'm running only 1 worker with concurrency of 4, I think this configuration alone can help me in achieving what i want. I just need to feel all of them at once to the queue.
– DUDE_MXP
Nov 17 '18 at 16:14
add a comment |
Don't chain the tasks together if they are independent. Adjust the settings on the celery worker for a specific queue to limit the number of workers to 4 using --concurrency 4 or -c 4.
answered Nov 17 '18 at 16:03
schillingtschillingt
5,30211722
Don't chain the tasks together if they are independent. Adjust the settings on the celery worker for a specific queue to limit the number of workers to 4 using --concurrency 4 or -c 4.
answered Nov 17 '18 at 16:03
schillingtschillingt
5,30211722
answered Nov 17 '18 at 16:03
schillingtschillingt
5,30211722
answered Nov 17 '18 at 16:03
schillingtschillingt
5,30211722
answered Nov 17 '18 at 16:03
schillingtschillingt
5,30211722
5,30211722
you mean that I feed all the 100 tasks to the queue, but since no of worker is 4 only 4 tasks will be processed at a time
– DUDE_MXP
Nov 17 '18 at 16:07
Basically, yes. You might have to tune some other settings, but that is the approach I would take.
– schillingt
Nov 17 '18 at 16:08
Right now i'm running only 1 worker with concurrency of 4, I think this configuration alone can help me in achieving what i want. I just need to feel all of them at once to the queue.
– DUDE_MXP
Nov 17 '18 at 16:14
add a comment |
you mean that I feed all the 100 tasks to the queue, but since no of worker is 4 only 4 tasks will be processed at a time
– DUDE_MXP
Nov 17 '18 at 16:07
Basically, yes. You might have to tune some other settings, but that is the approach I would take.
– schillingt
Nov 17 '18 at 16:08
Right now i'm running only 1 worker with concurrency of 4, I think this configuration alone can help me in achieving what i want. I just need to feel all of them at once to the queue.
– DUDE_MXP
Nov 17 '18 at 16:14
you mean that I feed all the 100 tasks to the queue, but since no of worker is 4 only 4 tasks will be processed at a time
– DUDE_MXP
Nov 17 '18 at 16:07
you mean that I feed all the 100 tasks to the queue, but since no of worker is 4 only 4 tasks will be processed at a time
– DUDE_MXP
Nov 17 '18 at 16:07
Basically, yes. You might have to tune some other settings, but that is the approach I would take.
– schillingt
Nov 17 '18 at 16:08
Basically, yes. You might have to tune some other settings, but that is the approach I would take.
– schillingt
Nov 17 '18 at 16:08
Right now i'm running only 1 worker with concurrency of 4, I think this configuration alone can help me in achieving what i want. I just need to feel all of them at once to the queue.
– DUDE_MXP
Nov 17 '18 at 16:14
Right now i'm running only 1 worker with concurrency of 4, I think this configuration alone can help me in achieving what i want. I just need to feel all of them at once to the queue.
– DUDE_MXP
Nov 17 '18 at 16:14
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53352122%2ferror-handling-while-using-chain-of-groups-in-celery%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Do the next set of tasks require the first set of tasks to be run?
– schillingt
Nov 17 '18 at 15:53
@schillingt no it's not required. All tasks are independent of each other.
– DUDE_MXP
Nov 17 '18 at 16:00