Producer throughput with varying acks=0,1,-1
I have been doing some performance tests with kafka cluster for my project. I have a question regarding the send call and the 'acks' property of producer. I observed below numbers with below invocation of send call. This is a simple fire and forget call.
producer.send(record); // fire and forget call
The topic has 5 partitions and I see below results with different acks value and replication factor. The kafka cluster has 5 nodes running with default values and using local disk
acks Replication factor=1 Replication factor=3
0 1330k msgs/sec 1260k msgs/sec
1 1220k msgs/sec 1200k msgs/sec
-1(all) 1220k msgs/sec 325k msgs/sec
As you can see as the acks value changes from 0 to all, the producer throughput decreases. What I am not able to understand is that if the producer send call is fire and forget in nature (see above) and producer is not waiting for any acknowledgements then why does the producer throughput drops as we move to stronger acks guarantees?
Any insights into how acks and producer send call works internally in Kakfa would be greatly appreciated.
P.S. I had asked this on kafka users mailing list but didn't get a reply so asking this on SO.
java apache-kafka kafka-producer-api
edited Nov 19 '18 at 10:56
Mike
2,0051722
asked Nov 19 '18 at 9:32
xabhixabhi
15817
add a comment |
I have been doing some performance tests with kafka cluster for my project. I have a question regarding the send call and the 'acks' property of producer. I observed below numbers with below invocation of send call. This is a simple fire and forget call.
producer.send(record); // fire and forget call
The topic has 5 partitions and I see below results with different acks value and replication factor. The kafka cluster has 5 nodes running with default values and using local disk
acks Replication factor=1 Replication factor=3
0 1330k msgs/sec 1260k msgs/sec
1 1220k msgs/sec 1200k msgs/sec
-1(all) 1220k msgs/sec 325k msgs/sec
As you can see as the acks value changes from 0 to all, the producer throughput decreases. What I am not able to understand is that if the producer send call is fire and forget in nature (see above) and producer is not waiting for any acknowledgements then why does the producer throughput drops as we move to stronger acks guarantees?
Any insights into how acks and producer send call works internally in Kakfa would be greatly appreciated.
P.S. I had asked this on kafka users mailing list but didn't get a reply so asking this on SO.
java apache-kafka kafka-producer-api
edited Nov 19 '18 at 10:56
Mike
2,0051722
asked Nov 19 '18 at 9:32
xabhixabhi
15817
add a comment |
I have been doing some performance tests with kafka cluster for my project. I have a question regarding the send call and the 'acks' property of producer. I observed below numbers with below invocation of send call. This is a simple fire and forget call.
producer.send(record); // fire and forget call
The topic has 5 partitions and I see below results with different acks value and replication factor. The kafka cluster has 5 nodes running with default values and using local disk
acks Replication factor=1 Replication factor=3
0 1330k msgs/sec 1260k msgs/sec
1 1220k msgs/sec 1200k msgs/sec
-1(all) 1220k msgs/sec 325k msgs/sec
As you can see as the acks value changes from 0 to all, the producer throughput decreases. What I am not able to understand is that if the producer send call is fire and forget in nature (see above) and producer is not waiting for any acknowledgements then why does the producer throughput drops as we move to stronger acks guarantees?
Any insights into how acks and producer send call works internally in Kakfa would be greatly appreciated.
P.S. I had asked this on kafka users mailing list but didn't get a reply so asking this on SO.
java apache-kafka kafka-producer-api
edited Nov 19 '18 at 10:56
Mike
2,0051722
asked Nov 19 '18 at 9:32
xabhixabhi
15817
I have been doing some performance tests with kafka cluster for my project. I have a question regarding the send call and the 'acks' property of producer. I observed below numbers with below invocation of send call. This is a simple fire and forget call.
producer.send(record); // fire and forget call
The topic has 5 partitions and I see below results with different acks value and replication factor. The kafka cluster has 5 nodes running with default values and using local disk
acks Replication factor=1 Replication factor=3
0 1330k msgs/sec 1260k msgs/sec
1 1220k msgs/sec 1200k msgs/sec
-1(all) 1220k msgs/sec 325k msgs/sec
As you can see as the acks value changes from 0 to all, the producer throughput decreases. What I am not able to understand is that if the producer send call is fire and forget in nature (see above) and producer is not waiting for any acknowledgements then why does the producer throughput drops as we move to stronger acks guarantees?
Any insights into how acks and producer send call works internally in Kakfa would be greatly appreciated.
P.S. I had asked this on kafka users mailing list but didn't get a reply so asking this on SO.
java apache-kafka kafka-producer-api
java apache-kafka kafka-producer-api
edited Nov 19 '18 at 10:56
Mike
2,0051722
asked Nov 19 '18 at 9:32
xabhixabhi
15817
edited Nov 19 '18 at 10:56
Mike
2,0051722
asked Nov 19 '18 at 9:32
xabhixabhi
15817
edited Nov 19 '18 at 10:56
Mike
2,0051722
edited Nov 19 '18 at 10:56
Mike
2,0051722
edited Nov 19 '18 at 10:56
Mike
2,0051722
2,0051722
asked Nov 19 '18 at 9:32
xabhixabhi
15817
asked Nov 19 '18 at 9:32
xabhixabhi
15817
asked Nov 19 '18 at 9:32
xabhixabhi
15817
15817
add a comment |
add a comment |
2 Answers
2
active
oldest
votes
The fact that you haven't a callback in the send method doesn't mean that it's fire and forget at the underlying level.
You have configured the producer with 3 different levels of ack which determine the "fire and forget" status or not.
With acks = 0, it means that the producer send the message but doesn't wait for any acks from the broker; it's the real "fire and forget". So as you can see it provides the higher throughput.
With acks = 1, the producer waits for the ack. This ack is sent by the broker (to which the producer is connected and that hosts the leader replica). It's not "fire and forget" of course.
With acks = -1, the producer waits for the ack. This ack is sent by the broker as above but only after having the messages replicated to all the replica followers on the other brokers. Of course in this case the throughput decrease if you increment the replication factor, because the message needs to be copied by more brokers (min.insync.replicas) before the "leader" broker returns back the ack to the producer.
Notice that with replication factor = 1, the ack = 1 and ack = -1 has same throughput because there is just one replica (the leader) so no need to copy to followers.
answered Nov 19 '18 at 19:54
ppatiernoppatierno
4,37211026
I also don't get it. send method javadocs says 'Asynchronously send a record to a topic...', how does it wait internally to support acks=all guarantee?
– freakman
Nov 20 '18 at 8:04
The producer as an internal buffer because sending messages works in batch. The batch with more messages is sent when a certain time (linger.ms) is elapsed or a specific size is reached (batch.size). From a client perspective is always async because the message goes just into the producer buffer which will send the message.
– ppatierno
Nov 20 '18 at 8:16
add a comment |
This is something about how kafka handling the produce request.
First, KafkaProducer.send is async by default. KafkaProducer has taken the heavily working for batch your produce request and send to broker. The broker will acking with produce response which in turn need wait for the min.insync.replicas from the remote followers. That's the reason.
answered Nov 19 '18 at 9:51
Terence Yi ZXTerence Yi ZX
11
If the send call is async and returns immediately then how does the acks affect it? And if producer waits for response then how is it async?
– xabhi
Nov 19 '18 at 20:24
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53371727%2fproducer-throughput-with-varying-acks-0-1-1%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
2 Answers
2
active
oldest
votes
2 Answers
2
active
oldest
votes
active
oldest
votes
active
oldest
votes
The fact that you haven't a callback in the send method doesn't mean that it's fire and forget at the underlying level.
You have configured the producer with 3 different levels of ack which determine the "fire and forget" status or not.
With acks = 0, it means that the producer send the message but doesn't wait for any acks from the broker; it's the real "fire and forget". So as you can see it provides the higher throughput.
With acks = 1, the producer waits for the ack. This ack is sent by the broker (to which the producer is connected and that hosts the leader replica). It's not "fire and forget" of course.
With acks = -1, the producer waits for the ack. This ack is sent by the broker as above but only after having the messages replicated to all the replica followers on the other brokers. Of course in this case the throughput decrease if you increment the replication factor, because the message needs to be copied by more brokers (min.insync.replicas) before the "leader" broker returns back the ack to the producer.
Notice that with replication factor = 1, the ack = 1 and ack = -1 has same throughput because there is just one replica (the leader) so no need to copy to followers.
answered Nov 19 '18 at 19:54
ppatiernoppatierno
4,37211026
I also don't get it. send method javadocs says 'Asynchronously send a record to a topic...', how does it wait internally to support acks=all guarantee?
– freakman
Nov 20 '18 at 8:04
The producer as an internal buffer because sending messages works in batch. The batch with more messages is sent when a certain time (linger.ms) is elapsed or a specific size is reached (batch.size). From a client perspective is always async because the message goes just into the producer buffer which will send the message.
– ppatierno
Nov 20 '18 at 8:16
add a comment |
The fact that you haven't a callback in the send method doesn't mean that it's fire and forget at the underlying level.
You have configured the producer with 3 different levels of ack which determine the "fire and forget" status or not.
With acks = 0, it means that the producer send the message but doesn't wait for any acks from the broker; it's the real "fire and forget". So as you can see it provides the higher throughput.
With acks = 1, the producer waits for the ack. This ack is sent by the broker (to which the producer is connected and that hosts the leader replica). It's not "fire and forget" of course.
With acks = -1, the producer waits for the ack. This ack is sent by the broker as above but only after having the messages replicated to all the replica followers on the other brokers. Of course in this case the throughput decrease if you increment the replication factor, because the message needs to be copied by more brokers (min.insync.replicas) before the "leader" broker returns back the ack to the producer.
Notice that with replication factor = 1, the ack = 1 and ack = -1 has same throughput because there is just one replica (the leader) so no need to copy to followers.
answered Nov 19 '18 at 19:54
ppatiernoppatierno
4,37211026
I also don't get it. send method javadocs says 'Asynchronously send a record to a topic...', how does it wait internally to support acks=all guarantee?
– freakman
Nov 20 '18 at 8:04
The producer as an internal buffer because sending messages works in batch. The batch with more messages is sent when a certain time (linger.ms) is elapsed or a specific size is reached (batch.size). From a client perspective is always async because the message goes just into the producer buffer which will send the message.
– ppatierno
Nov 20 '18 at 8:16
add a comment |
The fact that you haven't a callback in the send method doesn't mean that it's fire and forget at the underlying level.
You have configured the producer with 3 different levels of ack which determine the "fire and forget" status or not.
With acks = 0, it means that the producer send the message but doesn't wait for any acks from the broker; it's the real "fire and forget". So as you can see it provides the higher throughput.
With acks = 1, the producer waits for the ack. This ack is sent by the broker (to which the producer is connected and that hosts the leader replica). It's not "fire and forget" of course.
With acks = -1, the producer waits for the ack. This ack is sent by the broker as above but only after having the messages replicated to all the replica followers on the other brokers. Of course in this case the throughput decrease if you increment the replication factor, because the message needs to be copied by more brokers (min.insync.replicas) before the "leader" broker returns back the ack to the producer.
Notice that with replication factor = 1, the ack = 1 and ack = -1 has same throughput because there is just one replica (the leader) so no need to copy to followers.
answered Nov 19 '18 at 19:54
ppatiernoppatierno
4,37211026
The fact that you haven't a callback in the send method doesn't mean that it's fire and forget at the underlying level.
You have configured the producer with 3 different levels of ack which determine the "fire and forget" status or not.
With acks = 0, it means that the producer send the message but doesn't wait for any acks from the broker; it's the real "fire and forget". So as you can see it provides the higher throughput.
With acks = 1, the producer waits for the ack. This ack is sent by the broker (to which the producer is connected and that hosts the leader replica). It's not "fire and forget" of course.
With acks = -1, the producer waits for the ack. This ack is sent by the broker as above but only after having the messages replicated to all the replica followers on the other brokers. Of course in this case the throughput decrease if you increment the replication factor, because the message needs to be copied by more brokers (min.insync.replicas) before the "leader" broker returns back the ack to the producer.
Notice that with replication factor = 1, the ack = 1 and ack = -1 has same throughput because there is just one replica (the leader) so no need to copy to followers.
answered Nov 19 '18 at 19:54
ppatiernoppatierno
4,37211026
answered Nov 19 '18 at 19:54
ppatiernoppatierno
4,37211026
answered Nov 19 '18 at 19:54
ppatiernoppatierno
4,37211026
answered Nov 19 '18 at 19:54
ppatiernoppatierno
4,37211026
4,37211026
I also don't get it. send method javadocs says 'Asynchronously send a record to a topic...', how does it wait internally to support acks=all guarantee?
– freakman
Nov 20 '18 at 8:04
The producer as an internal buffer because sending messages works in batch. The batch with more messages is sent when a certain time (linger.ms) is elapsed or a specific size is reached (batch.size). From a client perspective is always async because the message goes just into the producer buffer which will send the message.
– ppatierno
Nov 20 '18 at 8:16
add a comment |
I also don't get it. send method javadocs says 'Asynchronously send a record to a topic...', how does it wait internally to support acks=all guarantee?
– freakman
Nov 20 '18 at 8:04
The producer as an internal buffer because sending messages works in batch. The batch with more messages is sent when a certain time (linger.ms) is elapsed or a specific size is reached (batch.size). From a client perspective is always async because the message goes just into the producer buffer which will send the message.
– ppatierno
Nov 20 '18 at 8:16
I also don't get it. send method javadocs says 'Asynchronously send a record to a topic...', how does it wait internally to support acks=all guarantee?
– freakman
Nov 20 '18 at 8:04
I also don't get it. send method javadocs says 'Asynchronously send a record to a topic...', how does it wait internally to support acks=all guarantee?
– freakman
Nov 20 '18 at 8:04
The producer as an internal buffer because sending messages works in batch. The batch with more messages is sent when a certain time (linger.ms) is elapsed or a specific size is reached (batch.size). From a client perspective is always async because the message goes just into the producer buffer which will send the message.
– ppatierno
Nov 20 '18 at 8:16
The producer as an internal buffer because sending messages works in batch. The batch with more messages is sent when a certain time (linger.ms) is elapsed or a specific size is reached (batch.size). From a client perspective is always async because the message goes just into the producer buffer which will send the message.
– ppatierno
Nov 20 '18 at 8:16
add a comment |
This is something about how kafka handling the produce request.
First, KafkaProducer.send is async by default. KafkaProducer has taken the heavily working for batch your produce request and send to broker. The broker will acking with produce response which in turn need wait for the min.insync.replicas from the remote followers. That's the reason.
answered Nov 19 '18 at 9:51
Terence Yi ZXTerence Yi ZX
11
If the send call is async and returns immediately then how does the acks affect it? And if producer waits for response then how is it async?
– xabhi
Nov 19 '18 at 20:24
add a comment |
This is something about how kafka handling the produce request.
First, KafkaProducer.send is async by default. KafkaProducer has taken the heavily working for batch your produce request and send to broker. The broker will acking with produce response which in turn need wait for the min.insync.replicas from the remote followers. That's the reason.
answered Nov 19 '18 at 9:51
Terence Yi ZXTerence Yi ZX
11
If the send call is async and returns immediately then how does the acks affect it? And if producer waits for response then how is it async?
– xabhi
Nov 19 '18 at 20:24
add a comment |
This is something about how kafka handling the produce request.
First, KafkaProducer.send is async by default. KafkaProducer has taken the heavily working for batch your produce request and send to broker. The broker will acking with produce response which in turn need wait for the min.insync.replicas from the remote followers. That's the reason.
answered Nov 19 '18 at 9:51
Terence Yi ZXTerence Yi ZX
11
This is something about how kafka handling the produce request.
First, KafkaProducer.send is async by default. KafkaProducer has taken the heavily working for batch your produce request and send to broker. The broker will acking with produce response which in turn need wait for the min.insync.replicas from the remote followers. That's the reason.
answered Nov 19 '18 at 9:51
Terence Yi ZXTerence Yi ZX
11
answered Nov 19 '18 at 9:51
Terence Yi ZXTerence Yi ZX
11
answered Nov 19 '18 at 9:51
Terence Yi ZXTerence Yi ZX
11
answered Nov 19 '18 at 9:51
Terence Yi ZXTerence Yi ZX
11
11
If the send call is async and returns immediately then how does the acks affect it? And if producer waits for response then how is it async?
– xabhi
Nov 19 '18 at 20:24
add a comment |
If the send call is async and returns immediately then how does the acks affect it? And if producer waits for response then how is it async?
– xabhi
Nov 19 '18 at 20:24
If the send call is async and returns immediately then how does the acks affect it? And if producer waits for response then how is it async?
– xabhi
Nov 19 '18 at 20:24
If the send call is async and returns immediately then how does the acks affect it? And if producer waits for response then how is it async?
– xabhi
Nov 19 '18 at 20:24
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53371727%2fproducer-throughput-with-varying-acks-0-1-1%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
