Kafka compaction on a topic stopped
.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty{ height:90px;width:728px;box-sizing:border-box;
}
I have multiple Kafka topics on which compaction is enabled. On all the topics compaction is happening fine but on one topic I am facing below issue.
The compaction happened until a point of time, after that it never happened. When I went through log cleaner logs there weren't any exceptions or logs telling me the potential reason why compaction stopped.
I am wondering what could be the reason for this behavior. How do i force my compaction to happen?
Any sort of advice would be helpful.
apache-kafka
edited Nov 21 '18 at 21:23
Mickael Maison
8,55342831
asked Nov 21 '18 at 21:07
user304611user304611
4619
add a comment |
I have multiple Kafka topics on which compaction is enabled. On all the topics compaction is happening fine but on one topic I am facing below issue.
The compaction happened until a point of time, after that it never happened. When I went through log cleaner logs there weren't any exceptions or logs telling me the potential reason why compaction stopped.
I am wondering what could be the reason for this behavior. How do i force my compaction to happen?
Any sort of advice would be helpful.
apache-kafka
edited Nov 21 '18 at 21:23
Mickael Maison
8,55342831
asked Nov 21 '18 at 21:07
user304611user304611
4619
How do you know compaction has stopped?
– Mickael Maison
Nov 21 '18 at 21:10
My disk size is growing more than it should, and from the logs I could see compaction is not happening.
– user304611
Nov 21 '18 at 21:13
add a comment |
I have multiple Kafka topics on which compaction is enabled. On all the topics compaction is happening fine but on one topic I am facing below issue.
The compaction happened until a point of time, after that it never happened. When I went through log cleaner logs there weren't any exceptions or logs telling me the potential reason why compaction stopped.
I am wondering what could be the reason for this behavior. How do i force my compaction to happen?
Any sort of advice would be helpful.
apache-kafka
edited Nov 21 '18 at 21:23
Mickael Maison
8,55342831
asked Nov 21 '18 at 21:07
user304611user304611
4619
I have multiple Kafka topics on which compaction is enabled. On all the topics compaction is happening fine but on one topic I am facing below issue.
The compaction happened until a point of time, after that it never happened. When I went through log cleaner logs there weren't any exceptions or logs telling me the potential reason why compaction stopped.
I am wondering what could be the reason for this behavior. How do i force my compaction to happen?
Any sort of advice would be helpful.
apache-kafka
apache-kafka
edited Nov 21 '18 at 21:23
Mickael Maison
8,55342831
asked Nov 21 '18 at 21:07
user304611user304611
4619
edited Nov 21 '18 at 21:23
Mickael Maison
8,55342831
asked Nov 21 '18 at 21:07
user304611user304611
4619
edited Nov 21 '18 at 21:23
Mickael Maison
8,55342831
edited Nov 21 '18 at 21:23
Mickael Maison
8,55342831
edited Nov 21 '18 at 21:23
Mickael Maison
8,55342831
8,55342831
asked Nov 21 '18 at 21:07
user304611user304611
4619
asked Nov 21 '18 at 21:07
user304611user304611
4619
asked Nov 21 '18 at 21:07
user304611user304611
4619
4619
How do you know compaction has stopped?
– Mickael Maison
Nov 21 '18 at 21:10
My disk size is growing more than it should, and from the logs I could see compaction is not happening.
– user304611
Nov 21 '18 at 21:13
add a comment |
How do you know compaction has stopped?
– Mickael Maison
Nov 21 '18 at 21:10
My disk size is growing more than it should, and from the logs I could see compaction is not happening.
– user304611
Nov 21 '18 at 21:13
How do you know compaction has stopped?
– Mickael Maison
Nov 21 '18 at 21:10
How do you know compaction has stopped?
– Mickael Maison
Nov 21 '18 at 21:10
My disk size is growing more than it should, and from the logs I could see compaction is not happening.
– user304611
Nov 21 '18 at 21:13
My disk size is growing more than it should, and from the logs I could see compaction is not happening.
– user304611
Nov 21 '18 at 21:13
add a comment |
1 Answer
1
active
oldest
votes
This usually happen because the compacting thread (log cleaner) crashed. Next step in troubleshooting is to find the reason it crashed and fix it. I'd search the main server logs for exceptions.
Log cleaner became a lot more stable over time, so if you are running versions older than 2.0, I recommend upgrading.
answered Nov 21 '18 at 21:21
Gwen ShapiraGwen Shapira
2,7821318
I am running version above 2, i have gone through all the server logs, nothing regarding any exceptions is found. Are the log cleaner threads separate for every topic? Could i force compact somehow?
– user304611
Nov 22 '18 at 3:33
All the new index, timestamp index files are getting created for new segments, but the process after that compaction is not happening.
– user304611
Nov 22 '18 at 4:12
@user306log-cleaner.logis separate fromserver.log, yes... But what do you mean "version above 2"? That's the current, and highest version of Kafka
– cricket_007
Nov 22 '18 at 20:23
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53420484%2fkafka-compaction-on-a-topic-stopped%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
This usually happen because the compacting thread (log cleaner) crashed. Next step in troubleshooting is to find the reason it crashed and fix it. I'd search the main server logs for exceptions.
Log cleaner became a lot more stable over time, so if you are running versions older than 2.0, I recommend upgrading.
answered Nov 21 '18 at 21:21
Gwen ShapiraGwen Shapira
2,7821318
I am running version above 2, i have gone through all the server logs, nothing regarding any exceptions is found. Are the log cleaner threads separate for every topic? Could i force compact somehow?
– user304611
Nov 22 '18 at 3:33
All the new index, timestamp index files are getting created for new segments, but the process after that compaction is not happening.
– user304611
Nov 22 '18 at 4:12
@user306log-cleaner.logis separate fromserver.log, yes... But what do you mean "version above 2"? That's the current, and highest version of Kafka
– cricket_007
Nov 22 '18 at 20:23
add a comment |
This usually happen because the compacting thread (log cleaner) crashed. Next step in troubleshooting is to find the reason it crashed and fix it. I'd search the main server logs for exceptions.
Log cleaner became a lot more stable over time, so if you are running versions older than 2.0, I recommend upgrading.
answered Nov 21 '18 at 21:21
Gwen ShapiraGwen Shapira
2,7821318
I am running version above 2, i have gone through all the server logs, nothing regarding any exceptions is found. Are the log cleaner threads separate for every topic? Could i force compact somehow?
– user304611
Nov 22 '18 at 3:33
All the new index, timestamp index files are getting created for new segments, but the process after that compaction is not happening.
– user304611
Nov 22 '18 at 4:12
@user306log-cleaner.logis separate fromserver.log, yes... But what do you mean "version above 2"? That's the current, and highest version of Kafka
– cricket_007
Nov 22 '18 at 20:23
add a comment |
This usually happen because the compacting thread (log cleaner) crashed. Next step in troubleshooting is to find the reason it crashed and fix it. I'd search the main server logs for exceptions.
Log cleaner became a lot more stable over time, so if you are running versions older than 2.0, I recommend upgrading.
answered Nov 21 '18 at 21:21
Gwen ShapiraGwen Shapira
2,7821318
This usually happen because the compacting thread (log cleaner) crashed. Next step in troubleshooting is to find the reason it crashed and fix it. I'd search the main server logs for exceptions.
Log cleaner became a lot more stable over time, so if you are running versions older than 2.0, I recommend upgrading.
answered Nov 21 '18 at 21:21
Gwen ShapiraGwen Shapira
2,7821318
answered Nov 21 '18 at 21:21
Gwen ShapiraGwen Shapira
2,7821318
answered Nov 21 '18 at 21:21
Gwen ShapiraGwen Shapira
2,7821318
answered Nov 21 '18 at 21:21
Gwen ShapiraGwen Shapira
2,7821318
2,7821318
I am running version above 2, i have gone through all the server logs, nothing regarding any exceptions is found. Are the log cleaner threads separate for every topic? Could i force compact somehow?
– user304611
Nov 22 '18 at 3:33
All the new index, timestamp index files are getting created for new segments, but the process after that compaction is not happening.
– user304611
Nov 22 '18 at 4:12
@user306log-cleaner.logis separate fromserver.log, yes... But what do you mean "version above 2"? That's the current, and highest version of Kafka
– cricket_007
Nov 22 '18 at 20:23
add a comment |
I am running version above 2, i have gone through all the server logs, nothing regarding any exceptions is found. Are the log cleaner threads separate for every topic? Could i force compact somehow?
– user304611
Nov 22 '18 at 3:33
All the new index, timestamp index files are getting created for new segments, but the process after that compaction is not happening.
– user304611
Nov 22 '18 at 4:12
@user306log-cleaner.logis separate fromserver.log, yes... But what do you mean "version above 2"? That's the current, and highest version of Kafka
– cricket_007
Nov 22 '18 at 20:23
I am running version above 2, i have gone through all the server logs, nothing regarding any exceptions is found. Are the log cleaner threads separate for every topic? Could i force compact somehow?
– user304611
Nov 22 '18 at 3:33
I am running version above 2, i have gone through all the server logs, nothing regarding any exceptions is found. Are the log cleaner threads separate for every topic? Could i force compact somehow?
– user304611
Nov 22 '18 at 3:33
All the new index, timestamp index files are getting created for new segments, but the process after that compaction is not happening.
– user304611
Nov 22 '18 at 4:12
All the new index, timestamp index files are getting created for new segments, but the process after that compaction is not happening.
– user304611
Nov 22 '18 at 4:12
@user306
log-cleaner.log is separate from server.log, yes... But what do you mean "version above 2"? That's the current, and highest version of Kafka– cricket_007
Nov 22 '18 at 20:23
@user306
log-cleaner.log is separate from server.log, yes... But what do you mean "version above 2"? That's the current, and highest version of Kafka– cricket_007
Nov 22 '18 at 20:23
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53420484%2fkafka-compaction-on-a-topic-stopped%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
How do you know compaction has stopped?
– Mickael Maison
Nov 21 '18 at 21:10
My disk size is growing more than it should, and from the logs I could see compaction is not happening.
– user304611
Nov 21 '18 at 21:13