apache kafka retention kstream and linked topic
I have a ktable kafka linked to a topic however when retention is done on the topic the messages are also deleted in my ktable And is it possible to keep the values of the Ktable while those of its topic are deleted?
apache-kafka apache-kafka-streams
edited Nov 14 '18 at 15:47
cricket_007
79.4k1142109
asked Nov 14 '18 at 10:11
Mlamlu
11
add a comment |
I have a ktable kafka linked to a topic however when retention is done on the topic the messages are also deleted in my ktable And is it possible to keep the values of the Ktable while those of its topic are deleted?
apache-kafka apache-kafka-streams
edited Nov 14 '18 at 15:47
cricket_007
79.4k1142109
asked Nov 14 '18 at 10:11
Mlamlu
11
Have you heard about topic compaction?
– cricket_007
Nov 14 '18 at 15:47
when retention is done on the topic the messages are also deleted in my ktableif the topic user retention but not compaction, the data would not be deleted from the ktable automatically -- however, if you loose the ktable, the data is of course gone.
– Matthias J. Sax
Nov 14 '18 at 23:43
@cricket_007 No, but I think this is the solution to my problem thank you!
– Mlamlu
Nov 19 '18 at 9:55
add a comment |
I have a ktable kafka linked to a topic however when retention is done on the topic the messages are also deleted in my ktable And is it possible to keep the values of the Ktable while those of its topic are deleted?
apache-kafka apache-kafka-streams
edited Nov 14 '18 at 15:47
cricket_007
79.4k1142109
asked Nov 14 '18 at 10:11
Mlamlu
11
I have a ktable kafka linked to a topic however when retention is done on the topic the messages are also deleted in my ktable And is it possible to keep the values of the Ktable while those of its topic are deleted?
apache-kafka apache-kafka-streams
apache-kafka apache-kafka-streams
edited Nov 14 '18 at 15:47
cricket_007
79.4k1142109
asked Nov 14 '18 at 10:11
Mlamlu
11
edited Nov 14 '18 at 15:47
cricket_007
79.4k1142109
asked Nov 14 '18 at 10:11
Mlamlu
11
edited Nov 14 '18 at 15:47
cricket_007
79.4k1142109
edited Nov 14 '18 at 15:47
cricket_007
79.4k1142109
edited Nov 14 '18 at 15:47
cricket_007
79.4k1142109
79.4k1142109
asked Nov 14 '18 at 10:11
Mlamlu
11
asked Nov 14 '18 at 10:11
Mlamlu
11
asked Nov 14 '18 at 10:11
Mlamlu
11
11
Have you heard about topic compaction?
– cricket_007
Nov 14 '18 at 15:47
when retention is done on the topic the messages are also deleted in my ktableif the topic user retention but not compaction, the data would not be deleted from the ktable automatically -- however, if you loose the ktable, the data is of course gone.
– Matthias J. Sax
Nov 14 '18 at 23:43
@cricket_007 No, but I think this is the solution to my problem thank you!
– Mlamlu
Nov 19 '18 at 9:55
add a comment |
Have you heard about topic compaction?
– cricket_007
Nov 14 '18 at 15:47
when retention is done on the topic the messages are also deleted in my ktableif the topic user retention but not compaction, the data would not be deleted from the ktable automatically -- however, if you loose the ktable, the data is of course gone.
– Matthias J. Sax
Nov 14 '18 at 23:43
@cricket_007 No, but I think this is the solution to my problem thank you!
– Mlamlu
Nov 19 '18 at 9:55
Have you heard about topic compaction?
– cricket_007
Nov 14 '18 at 15:47
Have you heard about topic compaction?
– cricket_007
Nov 14 '18 at 15:47
when retention is done on the topic the messages are also deleted in my ktable if the topic user retention but not compaction, the data would not be deleted from the ktable automatically -- however, if you loose the ktable, the data is of course gone.– Matthias J. Sax
Nov 14 '18 at 23:43
when retention is done on the topic the messages are also deleted in my ktable if the topic user retention but not compaction, the data would not be deleted from the ktable automatically -- however, if you loose the ktable, the data is of course gone.– Matthias J. Sax
Nov 14 '18 at 23:43
@cricket_007 No, but I think this is the solution to my problem thank you!
– Mlamlu
Nov 19 '18 at 9:55
@cricket_007 No, but I think this is the solution to my problem thank you!
– Mlamlu
Nov 19 '18 at 9:55
add a comment |
1 Answer
1
active
oldest
votes
Changelog topics are log compacted with infinite retention time. At any time, if there is a new event coming up your topic, it will update the state in the KTables for that key.
In your scenario, if you need to make the data available even when source topic is deleted, I will recommend publish the KTable into some output topic to make it more stable.
As KTables are exposed only within the applications and built on top of changelog topics, once your application is gone, you will lose the data unless you use persistent state store.
answered Nov 14 '18 at 15:56
Nishu Tayal
11.6k73481
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53297685%2fapache-kafka-retention-kstream-and-linked-topic%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
Changelog topics are log compacted with infinite retention time. At any time, if there is a new event coming up your topic, it will update the state in the KTables for that key.
In your scenario, if you need to make the data available even when source topic is deleted, I will recommend publish the KTable into some output topic to make it more stable.
As KTables are exposed only within the applications and built on top of changelog topics, once your application is gone, you will lose the data unless you use persistent state store.
answered Nov 14 '18 at 15:56
Nishu Tayal
11.6k73481
add a comment |
Changelog topics are log compacted with infinite retention time. At any time, if there is a new event coming up your topic, it will update the state in the KTables for that key.
In your scenario, if you need to make the data available even when source topic is deleted, I will recommend publish the KTable into some output topic to make it more stable.
As KTables are exposed only within the applications and built on top of changelog topics, once your application is gone, you will lose the data unless you use persistent state store.
answered Nov 14 '18 at 15:56
Nishu Tayal
11.6k73481
add a comment |
Changelog topics are log compacted with infinite retention time. At any time, if there is a new event coming up your topic, it will update the state in the KTables for that key.
In your scenario, if you need to make the data available even when source topic is deleted, I will recommend publish the KTable into some output topic to make it more stable.
As KTables are exposed only within the applications and built on top of changelog topics, once your application is gone, you will lose the data unless you use persistent state store.
answered Nov 14 '18 at 15:56
Nishu Tayal
11.6k73481
Changelog topics are log compacted with infinite retention time. At any time, if there is a new event coming up your topic, it will update the state in the KTables for that key.
In your scenario, if you need to make the data available even when source topic is deleted, I will recommend publish the KTable into some output topic to make it more stable.
As KTables are exposed only within the applications and built on top of changelog topics, once your application is gone, you will lose the data unless you use persistent state store.
answered Nov 14 '18 at 15:56
Nishu Tayal
11.6k73481
answered Nov 14 '18 at 15:56
Nishu Tayal
11.6k73481
answered Nov 14 '18 at 15:56
Nishu Tayal
11.6k73481
answered Nov 14 '18 at 15:56
Nishu Tayal
11.6k73481
11.6k73481
add a comment |
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Some of your past answers have not been well-received, and you're in danger of being blocked from answering.
Please pay close attention to the following guidance:
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53297685%2fapache-kafka-retention-kstream-and-linked-topic%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Have you heard about topic compaction?
– cricket_007
Nov 14 '18 at 15:47
when retention is done on the topic the messages are also deleted in my ktableif the topic user retention but not compaction, the data would not be deleted from the ktable automatically -- however, if you loose the ktable, the data is of course gone.– Matthias J. Sax
Nov 14 '18 at 23:43
@cricket_007 No, but I think this is the solution to my problem thank you!
– Mlamlu
Nov 19 '18 at 9:55