how to find out the produce order acked in the librdkafka library?
in librdkafka,
The data has been put in the transmission queue via following function.
virtual ErrorCode produce (Topic *topic, int32_t partition, int msgflags, void *payload, size_t len, const void *key, size_t key_len, void *msg_opaque)
The result of the transfer can be found in the dr_cb() registered as a callback through the poll(0).
class ExampleDeliveryReportCb : public RdKafka::DeliveryReportCb {
public:
void dr_cb (RdKafka::Message &message) {
// I want to know the order of produce.
}
};
If I produce 10 times, how can I know if the third produce has been successful? I can see the actual messages or the number of messages left in the transmission queue, but the result of third produce is unknown.
I want to synchronize the sequence number of the message produced and the sequence number of the message reported to be complete via dr_cb. What should I do?
apache-kafka producer-consumer kafka-producer-api
asked Nov 19 '18 at 13:36
김현우김현우
136
add a comment |
in librdkafka,
The data has been put in the transmission queue via following function.
virtual ErrorCode produce (Topic *topic, int32_t partition, int msgflags, void *payload, size_t len, const void *key, size_t key_len, void *msg_opaque)
The result of the transfer can be found in the dr_cb() registered as a callback through the poll(0).
class ExampleDeliveryReportCb : public RdKafka::DeliveryReportCb {
public:
void dr_cb (RdKafka::Message &message) {
// I want to know the order of produce.
}
};
If I produce 10 times, how can I know if the third produce has been successful? I can see the actual messages or the number of messages left in the transmission queue, but the result of third produce is unknown.
I want to synchronize the sequence number of the message produced and the sequence number of the message reported to be complete via dr_cb. What should I do?
apache-kafka producer-consumer kafka-producer-api
asked Nov 19 '18 at 13:36
김현우김현우
136
add a comment |
in librdkafka,
The data has been put in the transmission queue via following function.
virtual ErrorCode produce (Topic *topic, int32_t partition, int msgflags, void *payload, size_t len, const void *key, size_t key_len, void *msg_opaque)
The result of the transfer can be found in the dr_cb() registered as a callback through the poll(0).
class ExampleDeliveryReportCb : public RdKafka::DeliveryReportCb {
public:
void dr_cb (RdKafka::Message &message) {
// I want to know the order of produce.
}
};
If I produce 10 times, how can I know if the third produce has been successful? I can see the actual messages or the number of messages left in the transmission queue, but the result of third produce is unknown.
I want to synchronize the sequence number of the message produced and the sequence number of the message reported to be complete via dr_cb. What should I do?
apache-kafka producer-consumer kafka-producer-api
asked Nov 19 '18 at 13:36
김현우김현우
136
in librdkafka,
The data has been put in the transmission queue via following function.
virtual ErrorCode produce (Topic *topic, int32_t partition, int msgflags, void *payload, size_t len, const void *key, size_t key_len, void *msg_opaque)
The result of the transfer can be found in the dr_cb() registered as a callback through the poll(0).
class ExampleDeliveryReportCb : public RdKafka::DeliveryReportCb {
public:
void dr_cb (RdKafka::Message &message) {
// I want to know the order of produce.
}
};
If I produce 10 times, how can I know if the third produce has been successful? I can see the actual messages or the number of messages left in the transmission queue, but the result of third produce is unknown.
I want to synchronize the sequence number of the message produced and the sequence number of the message reported to be complete via dr_cb. What should I do?
apache-kafka producer-consumer kafka-producer-api
apache-kafka producer-consumer kafka-producer-api
asked Nov 19 '18 at 13:36
김현우김현우
136
asked Nov 19 '18 at 13:36
김현우김현우
136
asked Nov 19 '18 at 13:36
김현우김현우
136
asked Nov 19 '18 at 13:36
김현우김현우
136
asked Nov 19 '18 at 13:36
김현우김현우
136
136
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
Ordering is only a thing per partition in Kafka, if you produce to multiple partitions there is no ordering inbetween those partitions.
As for messages produced to the same partition they will be produced in their original order, unless there is an error which warrants a retry, in which case reordering is possible.
You can either set max.in.flight=1 to avoid ordering, which unfortunately also decreases throughput and increases latency, or use the upcoming Idempotent Producer (only on master branch until next release) with enable.idempotence=true to get guaranteed ordered delivery at very little throughput cost.
You can use the message opaque to attach a pointer to a message in produce() which you'll get back as msg_opaque in the delivery report, allowing you to map delivery reports to the original object you produced.
answered Nov 19 '18 at 16:35
EdenhillEdenhill
1,3121122
thanks! and There are a few additional questions. I use version 0.11.4. 1. Is the socket.max.fails option not related to TCP 3-way handshake? If I set this value to 10 and produce to a broker that does not have the kafka service enabled, it will take more than 10 times to send a syn packet and receive an rst packet.
– 김현우
Nov 19 '18 at 23:46
2. Is the request.required.acks option and message timeout irrelevant? If I set this value to 0 and then produce, the document states that no ack is received from the broker. That is, outq_len () should be decremented regardless of the success of the transmission, but outq_len () does not decrease, but rather causes a timeout. What am I missing?
– 김현우
Nov 19 '18 at 23:50
socket.max.fails operates on the application layer, namely it is a threshold of the number of failed Kafka protocol requests before the connection is torn down. Leave it at its default value.
– Edenhill
Nov 20 '18 at 9:23
If a connection to the broker can't be made the produced messages will eventually time out (message.timeout.ms). The producer can't even attempt to produce a message if the broker connection is down, regardless if acks=0.
– Edenhill
Nov 20 '18 at 9:24
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53375833%2fhow-to-find-out-the-produce-order-acked-in-the-librdkafka-library%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
Ordering is only a thing per partition in Kafka, if you produce to multiple partitions there is no ordering inbetween those partitions.
As for messages produced to the same partition they will be produced in their original order, unless there is an error which warrants a retry, in which case reordering is possible.
You can either set max.in.flight=1 to avoid ordering, which unfortunately also decreases throughput and increases latency, or use the upcoming Idempotent Producer (only on master branch until next release) with enable.idempotence=true to get guaranteed ordered delivery at very little throughput cost.
You can use the message opaque to attach a pointer to a message in produce() which you'll get back as msg_opaque in the delivery report, allowing you to map delivery reports to the original object you produced.
answered Nov 19 '18 at 16:35
EdenhillEdenhill
1,3121122
thanks! and There are a few additional questions. I use version 0.11.4. 1. Is the socket.max.fails option not related to TCP 3-way handshake? If I set this value to 10 and produce to a broker that does not have the kafka service enabled, it will take more than 10 times to send a syn packet and receive an rst packet.
– 김현우
Nov 19 '18 at 23:46
2. Is the request.required.acks option and message timeout irrelevant? If I set this value to 0 and then produce, the document states that no ack is received from the broker. That is, outq_len () should be decremented regardless of the success of the transmission, but outq_len () does not decrease, but rather causes a timeout. What am I missing?
– 김현우
Nov 19 '18 at 23:50
socket.max.fails operates on the application layer, namely it is a threshold of the number of failed Kafka protocol requests before the connection is torn down. Leave it at its default value.
– Edenhill
Nov 20 '18 at 9:23
If a connection to the broker can't be made the produced messages will eventually time out (message.timeout.ms). The producer can't even attempt to produce a message if the broker connection is down, regardless if acks=0.
– Edenhill
Nov 20 '18 at 9:24
add a comment |
Ordering is only a thing per partition in Kafka, if you produce to multiple partitions there is no ordering inbetween those partitions.
As for messages produced to the same partition they will be produced in their original order, unless there is an error which warrants a retry, in which case reordering is possible.
You can either set max.in.flight=1 to avoid ordering, which unfortunately also decreases throughput and increases latency, or use the upcoming Idempotent Producer (only on master branch until next release) with enable.idempotence=true to get guaranteed ordered delivery at very little throughput cost.
You can use the message opaque to attach a pointer to a message in produce() which you'll get back as msg_opaque in the delivery report, allowing you to map delivery reports to the original object you produced.
answered Nov 19 '18 at 16:35
EdenhillEdenhill
1,3121122
thanks! and There are a few additional questions. I use version 0.11.4. 1. Is the socket.max.fails option not related to TCP 3-way handshake? If I set this value to 10 and produce to a broker that does not have the kafka service enabled, it will take more than 10 times to send a syn packet and receive an rst packet.
– 김현우
Nov 19 '18 at 23:46
2. Is the request.required.acks option and message timeout irrelevant? If I set this value to 0 and then produce, the document states that no ack is received from the broker. That is, outq_len () should be decremented regardless of the success of the transmission, but outq_len () does not decrease, but rather causes a timeout. What am I missing?
– 김현우
Nov 19 '18 at 23:50
socket.max.fails operates on the application layer, namely it is a threshold of the number of failed Kafka protocol requests before the connection is torn down. Leave it at its default value.
– Edenhill
Nov 20 '18 at 9:23
If a connection to the broker can't be made the produced messages will eventually time out (message.timeout.ms). The producer can't even attempt to produce a message if the broker connection is down, regardless if acks=0.
– Edenhill
Nov 20 '18 at 9:24
add a comment |
Ordering is only a thing per partition in Kafka, if you produce to multiple partitions there is no ordering inbetween those partitions.
As for messages produced to the same partition they will be produced in their original order, unless there is an error which warrants a retry, in which case reordering is possible.
You can either set max.in.flight=1 to avoid ordering, which unfortunately also decreases throughput and increases latency, or use the upcoming Idempotent Producer (only on master branch until next release) with enable.idempotence=true to get guaranteed ordered delivery at very little throughput cost.
You can use the message opaque to attach a pointer to a message in produce() which you'll get back as msg_opaque in the delivery report, allowing you to map delivery reports to the original object you produced.
answered Nov 19 '18 at 16:35
EdenhillEdenhill
1,3121122
Ordering is only a thing per partition in Kafka, if you produce to multiple partitions there is no ordering inbetween those partitions.
As for messages produced to the same partition they will be produced in their original order, unless there is an error which warrants a retry, in which case reordering is possible.
You can either set max.in.flight=1 to avoid ordering, which unfortunately also decreases throughput and increases latency, or use the upcoming Idempotent Producer (only on master branch until next release) with enable.idempotence=true to get guaranteed ordered delivery at very little throughput cost.
You can use the message opaque to attach a pointer to a message in produce() which you'll get back as msg_opaque in the delivery report, allowing you to map delivery reports to the original object you produced.
answered Nov 19 '18 at 16:35
EdenhillEdenhill
1,3121122
answered Nov 19 '18 at 16:35
EdenhillEdenhill
1,3121122
answered Nov 19 '18 at 16:35
EdenhillEdenhill
1,3121122
answered Nov 19 '18 at 16:35
EdenhillEdenhill
1,3121122
1,3121122
thanks! and There are a few additional questions. I use version 0.11.4. 1. Is the socket.max.fails option not related to TCP 3-way handshake? If I set this value to 10 and produce to a broker that does not have the kafka service enabled, it will take more than 10 times to send a syn packet and receive an rst packet.
– 김현우
Nov 19 '18 at 23:46
2. Is the request.required.acks option and message timeout irrelevant? If I set this value to 0 and then produce, the document states that no ack is received from the broker. That is, outq_len () should be decremented regardless of the success of the transmission, but outq_len () does not decrease, but rather causes a timeout. What am I missing?
– 김현우
Nov 19 '18 at 23:50
socket.max.fails operates on the application layer, namely it is a threshold of the number of failed Kafka protocol requests before the connection is torn down. Leave it at its default value.
– Edenhill
Nov 20 '18 at 9:23
If a connection to the broker can't be made the produced messages will eventually time out (message.timeout.ms). The producer can't even attempt to produce a message if the broker connection is down, regardless if acks=0.
– Edenhill
Nov 20 '18 at 9:24
add a comment |
thanks! and There are a few additional questions. I use version 0.11.4. 1. Is the socket.max.fails option not related to TCP 3-way handshake? If I set this value to 10 and produce to a broker that does not have the kafka service enabled, it will take more than 10 times to send a syn packet and receive an rst packet.
– 김현우
Nov 19 '18 at 23:46
2. Is the request.required.acks option and message timeout irrelevant? If I set this value to 0 and then produce, the document states that no ack is received from the broker. That is, outq_len () should be decremented regardless of the success of the transmission, but outq_len () does not decrease, but rather causes a timeout. What am I missing?
– 김현우
Nov 19 '18 at 23:50
socket.max.fails operates on the application layer, namely it is a threshold of the number of failed Kafka protocol requests before the connection is torn down. Leave it at its default value.
– Edenhill
Nov 20 '18 at 9:23
If a connection to the broker can't be made the produced messages will eventually time out (message.timeout.ms). The producer can't even attempt to produce a message if the broker connection is down, regardless if acks=0.
– Edenhill
Nov 20 '18 at 9:24
thanks! and There are a few additional questions. I use version 0.11.4. 1. Is the socket.max.fails option not related to TCP 3-way handshake? If I set this value to 10 and produce to a broker that does not have the kafka service enabled, it will take more than 10 times to send a syn packet and receive an rst packet.
– 김현우
Nov 19 '18 at 23:46
thanks! and There are a few additional questions. I use version 0.11.4. 1. Is the socket.max.fails option not related to TCP 3-way handshake? If I set this value to 10 and produce to a broker that does not have the kafka service enabled, it will take more than 10 times to send a syn packet and receive an rst packet.
– 김현우
Nov 19 '18 at 23:46
2. Is the request.required.acks option and message timeout irrelevant? If I set this value to 0 and then produce, the document states that no ack is received from the broker. That is, outq_len () should be decremented regardless of the success of the transmission, but outq_len () does not decrease, but rather causes a timeout. What am I missing?
– 김현우
Nov 19 '18 at 23:50
2. Is the request.required.acks option and message timeout irrelevant? If I set this value to 0 and then produce, the document states that no ack is received from the broker. That is, outq_len () should be decremented regardless of the success of the transmission, but outq_len () does not decrease, but rather causes a timeout. What am I missing?
– 김현우
Nov 19 '18 at 23:50
socket.max.fails operates on the application layer, namely it is a threshold of the number of failed Kafka protocol requests before the connection is torn down. Leave it at its default value.
– Edenhill
Nov 20 '18 at 9:23
socket.max.fails operates on the application layer, namely it is a threshold of the number of failed Kafka protocol requests before the connection is torn down. Leave it at its default value.
– Edenhill
Nov 20 '18 at 9:23
If a connection to the broker can't be made the produced messages will eventually time out (message.timeout.ms). The producer can't even attempt to produce a message if the broker connection is down, regardless if acks=0.
– Edenhill
Nov 20 '18 at 9:24
If a connection to the broker can't be made the produced messages will eventually time out (message.timeout.ms). The producer can't even attempt to produce a message if the broker connection is down, regardless if acks=0.
– Edenhill
Nov 20 '18 at 9:24
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53375833%2fhow-to-find-out-the-produce-order-acked-in-the-librdkafka-library%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
