ECS Rolling Update : Healthy task killed
I have an existing service with 2 tasks running. I updated the existing service with a new task definition and the old one was dergistered. The new task failed to come up but the old tasks were killed and the service tried to bring up the new tasks but it failed. This leads to a situation when there are no tasks running at all.
The reason the new tasks were not able to come up was because of a bug in my code. But, shouldn't ECS have ensured that the old healthy tasks were still running when the new ones where not up.
When does ECS decides to kill the old tasks?
I understand it is when the task status goes to "Running". Does "Running" means, it is healthy? On further debugging, I found that the problematic task was in running state for some time.
amazon-web-services docker amazon-ecs
asked Nov 20 '18 at 12:16
Sunil Kumar MohantySunil Kumar Mohanty
5418
add a comment |
I have an existing service with 2 tasks running. I updated the existing service with a new task definition and the old one was dergistered. The new task failed to come up but the old tasks were killed and the service tried to bring up the new tasks but it failed. This leads to a situation when there are no tasks running at all.
The reason the new tasks were not able to come up was because of a bug in my code. But, shouldn't ECS have ensured that the old healthy tasks were still running when the new ones where not up.
When does ECS decides to kill the old tasks?
I understand it is when the task status goes to "Running". Does "Running" means, it is healthy? On further debugging, I found that the problematic task was in running state for some time.
amazon-web-services docker amazon-ecs
asked Nov 20 '18 at 12:16
Sunil Kumar MohantySunil Kumar Mohanty
5418
add a comment |
I have an existing service with 2 tasks running. I updated the existing service with a new task definition and the old one was dergistered. The new task failed to come up but the old tasks were killed and the service tried to bring up the new tasks but it failed. This leads to a situation when there are no tasks running at all.
The reason the new tasks were not able to come up was because of a bug in my code. But, shouldn't ECS have ensured that the old healthy tasks were still running when the new ones where not up.
When does ECS decides to kill the old tasks?
I understand it is when the task status goes to "Running". Does "Running" means, it is healthy? On further debugging, I found that the problematic task was in running state for some time.
amazon-web-services docker amazon-ecs
asked Nov 20 '18 at 12:16
Sunil Kumar MohantySunil Kumar Mohanty
5418
I have an existing service with 2 tasks running. I updated the existing service with a new task definition and the old one was dergistered. The new task failed to come up but the old tasks were killed and the service tried to bring up the new tasks but it failed. This leads to a situation when there are no tasks running at all.
The reason the new tasks were not able to come up was because of a bug in my code. But, shouldn't ECS have ensured that the old healthy tasks were still running when the new ones where not up.
When does ECS decides to kill the old tasks?
I understand it is when the task status goes to "Running". Does "Running" means, it is healthy? On further debugging, I found that the problematic task was in running state for some time.
amazon-web-services docker amazon-ecs
amazon-web-services docker amazon-ecs
asked Nov 20 '18 at 12:16
Sunil Kumar MohantySunil Kumar Mohanty
5418
asked Nov 20 '18 at 12:16
Sunil Kumar MohantySunil Kumar Mohanty
5418
edited Nov 20 '18 at 12:48
Sunil Kumar Mohanty
asked Nov 20 '18 at 12:16
Sunil Kumar MohantySunil Kumar Mohanty
5418
asked Nov 20 '18 at 12:16
Sunil Kumar MohantySunil Kumar Mohanty
5418
asked Nov 20 '18 at 12:16
Sunil Kumar MohantySunil Kumar Mohanty
5418
5418
add a comment |
add a comment |
2 Answers
2
active
oldest
votes
Well, let me explain what is going on:
When you update your Task Definition (of which you assumingly have some tasks running), ECS tries to execute a blue-green deployment, i.e. wait to have new tasks in ready state and then stop the old ones.
Now, your task will enter ready state when the container boots, but when it tries to execute some code which has an error inside, it will crash and die, so it will change its state again, but it has already passed through ready state and so the old tasks have already been killed.
answered Nov 20 '18 at 12:45
AlexKAlexK
869513
Thanks The service is behind ALB. Does it not check if the the tasks are healthy or not? If not, could you suggest a better way to handle this. In my case, the application takes almost 15 secs to come up and there is a time period where both the old and the new tasks are running but the new ones are not healthy (ALB health check would fail)
– Sunil Kumar Mohanty
Nov 20 '18 at 12:49
Refer to this stackoverflow.com/questions/51425983/…, I think it might help.
– AlexK
Nov 20 '18 at 12:53
Have you had any success?
– AlexK
Nov 20 '18 at 14:47
I had a discussion with the AWS team and figured that ECS will not kill old task if the new tasks are not considered healthy. When a service is tied with ALB, ECS will not deregister old tasks until new tasks are returned as healthy by ALB.
– Sunil Kumar Mohanty
Dec 7 '18 at 8:22
add a comment |
I had a discussion with the AWS team and figured that ECS will not kill old task if the new tasks are not considered healthy. When a service is tied with ALB, ECS will not deregister old tasks until new tasks are marked as healthy by ALB.
answered Dec 7 '18 at 8:23
Sunil Kumar MohantySunil Kumar Mohanty
5418
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53392805%2fecs-rolling-update-healthy-task-killed%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
2 Answers
2
active
oldest
votes
2 Answers
2
active
oldest
votes
active
oldest
votes
active
oldest
votes
Well, let me explain what is going on:
When you update your Task Definition (of which you assumingly have some tasks running), ECS tries to execute a blue-green deployment, i.e. wait to have new tasks in ready state and then stop the old ones.
Now, your task will enter ready state when the container boots, but when it tries to execute some code which has an error inside, it will crash and die, so it will change its state again, but it has already passed through ready state and so the old tasks have already been killed.
answered Nov 20 '18 at 12:45
AlexKAlexK
869513
Thanks The service is behind ALB. Does it not check if the the tasks are healthy or not? If not, could you suggest a better way to handle this. In my case, the application takes almost 15 secs to come up and there is a time period where both the old and the new tasks are running but the new ones are not healthy (ALB health check would fail)
– Sunil Kumar Mohanty
Nov 20 '18 at 12:49
Refer to this stackoverflow.com/questions/51425983/…, I think it might help.
– AlexK
Nov 20 '18 at 12:53
Have you had any success?
– AlexK
Nov 20 '18 at 14:47
I had a discussion with the AWS team and figured that ECS will not kill old task if the new tasks are not considered healthy. When a service is tied with ALB, ECS will not deregister old tasks until new tasks are returned as healthy by ALB.
– Sunil Kumar Mohanty
Dec 7 '18 at 8:22
add a comment |
Well, let me explain what is going on:
When you update your Task Definition (of which you assumingly have some tasks running), ECS tries to execute a blue-green deployment, i.e. wait to have new tasks in ready state and then stop the old ones.
Now, your task will enter ready state when the container boots, but when it tries to execute some code which has an error inside, it will crash and die, so it will change its state again, but it has already passed through ready state and so the old tasks have already been killed.
answered Nov 20 '18 at 12:45
AlexKAlexK
869513
Thanks The service is behind ALB. Does it not check if the the tasks are healthy or not? If not, could you suggest a better way to handle this. In my case, the application takes almost 15 secs to come up and there is a time period where both the old and the new tasks are running but the new ones are not healthy (ALB health check would fail)
– Sunil Kumar Mohanty
Nov 20 '18 at 12:49
Refer to this stackoverflow.com/questions/51425983/…, I think it might help.
– AlexK
Nov 20 '18 at 12:53
Have you had any success?
– AlexK
Nov 20 '18 at 14:47
I had a discussion with the AWS team and figured that ECS will not kill old task if the new tasks are not considered healthy. When a service is tied with ALB, ECS will not deregister old tasks until new tasks are returned as healthy by ALB.
– Sunil Kumar Mohanty
Dec 7 '18 at 8:22
add a comment |
Well, let me explain what is going on:
When you update your Task Definition (of which you assumingly have some tasks running), ECS tries to execute a blue-green deployment, i.e. wait to have new tasks in ready state and then stop the old ones.
Now, your task will enter ready state when the container boots, but when it tries to execute some code which has an error inside, it will crash and die, so it will change its state again, but it has already passed through ready state and so the old tasks have already been killed.
answered Nov 20 '18 at 12:45
AlexKAlexK
869513
Well, let me explain what is going on:
When you update your Task Definition (of which you assumingly have some tasks running), ECS tries to execute a blue-green deployment, i.e. wait to have new tasks in ready state and then stop the old ones.
Now, your task will enter ready state when the container boots, but when it tries to execute some code which has an error inside, it will crash and die, so it will change its state again, but it has already passed through ready state and so the old tasks have already been killed.
answered Nov 20 '18 at 12:45
AlexKAlexK
869513
edited Nov 20 '18 at 12:49
answered Nov 20 '18 at 12:45
AlexKAlexK
869513
answered Nov 20 '18 at 12:45
AlexKAlexK
869513
answered Nov 20 '18 at 12:45
AlexKAlexK
869513
869513
Thanks The service is behind ALB. Does it not check if the the tasks are healthy or not? If not, could you suggest a better way to handle this. In my case, the application takes almost 15 secs to come up and there is a time period where both the old and the new tasks are running but the new ones are not healthy (ALB health check would fail)
– Sunil Kumar Mohanty
Nov 20 '18 at 12:49
Refer to this stackoverflow.com/questions/51425983/…, I think it might help.
– AlexK
Nov 20 '18 at 12:53
Have you had any success?
– AlexK
Nov 20 '18 at 14:47
I had a discussion with the AWS team and figured that ECS will not kill old task if the new tasks are not considered healthy. When a service is tied with ALB, ECS will not deregister old tasks until new tasks are returned as healthy by ALB.
– Sunil Kumar Mohanty
Dec 7 '18 at 8:22
add a comment |
Thanks The service is behind ALB. Does it not check if the the tasks are healthy or not? If not, could you suggest a better way to handle this. In my case, the application takes almost 15 secs to come up and there is a time period where both the old and the new tasks are running but the new ones are not healthy (ALB health check would fail)
– Sunil Kumar Mohanty
Nov 20 '18 at 12:49
Refer to this stackoverflow.com/questions/51425983/…, I think it might help.
– AlexK
Nov 20 '18 at 12:53
Have you had any success?
– AlexK
Nov 20 '18 at 14:47
I had a discussion with the AWS team and figured that ECS will not kill old task if the new tasks are not considered healthy. When a service is tied with ALB, ECS will not deregister old tasks until new tasks are returned as healthy by ALB.
– Sunil Kumar Mohanty
Dec 7 '18 at 8:22
Thanks The service is behind ALB. Does it not check if the the tasks are healthy or not? If not, could you suggest a better way to handle this. In my case, the application takes almost 15 secs to come up and there is a time period where both the old and the new tasks are running but the new ones are not healthy (ALB health check would fail)
– Sunil Kumar Mohanty
Nov 20 '18 at 12:49
Thanks The service is behind ALB. Does it not check if the the tasks are healthy or not? If not, could you suggest a better way to handle this. In my case, the application takes almost 15 secs to come up and there is a time period where both the old and the new tasks are running but the new ones are not healthy (ALB health check would fail)
– Sunil Kumar Mohanty
Nov 20 '18 at 12:49
Refer to this stackoverflow.com/questions/51425983/…, I think it might help.
– AlexK
Nov 20 '18 at 12:53
Refer to this stackoverflow.com/questions/51425983/…, I think it might help.
– AlexK
Nov 20 '18 at 12:53
Have you had any success?
– AlexK
Nov 20 '18 at 14:47
Have you had any success?
– AlexK
Nov 20 '18 at 14:47
I had a discussion with the AWS team and figured that ECS will not kill old task if the new tasks are not considered healthy. When a service is tied with ALB, ECS will not deregister old tasks until new tasks are returned as healthy by ALB.
– Sunil Kumar Mohanty
Dec 7 '18 at 8:22
I had a discussion with the AWS team and figured that ECS will not kill old task if the new tasks are not considered healthy. When a service is tied with ALB, ECS will not deregister old tasks until new tasks are returned as healthy by ALB.
– Sunil Kumar Mohanty
Dec 7 '18 at 8:22
add a comment |
I had a discussion with the AWS team and figured that ECS will not kill old task if the new tasks are not considered healthy. When a service is tied with ALB, ECS will not deregister old tasks until new tasks are marked as healthy by ALB.
answered Dec 7 '18 at 8:23
Sunil Kumar MohantySunil Kumar Mohanty
5418
add a comment |
I had a discussion with the AWS team and figured that ECS will not kill old task if the new tasks are not considered healthy. When a service is tied with ALB, ECS will not deregister old tasks until new tasks are marked as healthy by ALB.
answered Dec 7 '18 at 8:23
Sunil Kumar MohantySunil Kumar Mohanty
5418
add a comment |
I had a discussion with the AWS team and figured that ECS will not kill old task if the new tasks are not considered healthy. When a service is tied with ALB, ECS will not deregister old tasks until new tasks are marked as healthy by ALB.
answered Dec 7 '18 at 8:23
Sunil Kumar MohantySunil Kumar Mohanty
5418
I had a discussion with the AWS team and figured that ECS will not kill old task if the new tasks are not considered healthy. When a service is tied with ALB, ECS will not deregister old tasks until new tasks are marked as healthy by ALB.
answered Dec 7 '18 at 8:23
Sunil Kumar MohantySunil Kumar Mohanty
5418
answered Dec 7 '18 at 8:23
Sunil Kumar MohantySunil Kumar Mohanty
5418
answered Dec 7 '18 at 8:23
Sunil Kumar MohantySunil Kumar Mohanty
5418
answered Dec 7 '18 at 8:23
Sunil Kumar MohantySunil Kumar Mohanty
5418
5418
add a comment |
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53392805%2fecs-rolling-update-healthy-task-killed%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
