Repository navigation
[adapters] Leave batching to librdkafka in the exactly-once Kafka output - #7432
Merged
Merged
Conversation
blp
approved these changes
Oct 12, 2026
blp
enabled auto-merge
October 12, 2026 22:52
auto-merge was automatically disabled
October 22, 2026 23:34
Head branch was pushed to by a user without write access
5 of 11 tasks
Contributor
|
Can you please rebase again? |
jacksehr
force-pushed
the
ft-kafka-producer-batching
branch
from
October 30, 2026 07:42
7a46727 to
60aa0e7
Compare
…put producer. The producer forced batch.size, batch.num.messages and linger.ms to their minimums, so every message went in its own request and capped the sink at about 400 messages/s. Leave the three settings at the librdkafka defaults. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Signed-off-by: Jack She <jackshe2702@gmail.com>
Add a changelog entry and a note in the Kafka sink docs that batching is left to librdkafka and can be tuned in fault_tolerance.producer_options. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Signed-off-by: Jack She <jackshe2702@gmail.com>
jacksehr
force-pushed
the
ft-kafka-producer-batching
branch
from
November 2, 2026 03:22
60aa0e7 to
33466dd
Compare
github-merge-queue
Bot
removed this pull request from the merge queue due to failed status checks
Nov 2, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Describe Manual Test Plan
We patched an enterprise image with the changes on this branch, deployed it to our staging cluster, and then imported 10M+ records with no issues in
exactly_oncemode.Worth noting that test was done with this change incorporated as well, which should be orthogonal.
Why
The exactly-once Kafka output forced
batch.size=1,batch.num.messages=1andlinger.ms=0, so every message went in its own produce request. That capped the sink at roughly 400 msg/s in our setup, and we found that large batches could not be processed before their Kafka transactions timed out.What
Stop setting the three options, so librdkafka's defaults apply, and
producer_optionscan tune them.commit_transactionstill flushes before the transaction ends, and recovery reads only the last committed key per partition, so neither depends on request boundaries.It's also worth noting that, while not covered in this PR, we observed that there's no abort and retry behaviour when a write via a Kafka transaction times out in
exactly-onceft mode.Checklist
In my opinion, passing the existing tests is sufficient here, but happy to add tests if needed.
Breaking Changes?
Describe Incompatible Changes
Default producer batching changes, and
batch.size,batch.num.messagesandlinger.msare no longer rejected when set inproducer_options.🤖 Code generated with Claude Code