← 输入框|github.com|刷新
Skip to content

[adapters] Leave batching to librdkafka in the exactly-once Kafka output - #7432

Merged
mihaibudiu merged 3 commits into
feldera:mainfrom
jacksehr:ft-kafka-producer-batching
Nov 3, 2026
Merged

mihaibudiu merged 3 commits into
feldera:mainfrom
jacksehr:ft-kafka-producer-batching

Conversation

@jacksehr

@jacksehr jacksehr commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

Describe Manual Test Plan

We patched an enterprise image with the changes on this branch, deployed it to our staging cluster, and then imported 10M+ records with no issues in exactly_once mode.

Worth noting that test was done with this change incorporated as well, which should be orthogonal.

Why

The exactly-once Kafka output forced batch.size=1, batch.num.messages=1 and linger.ms=0, so every message went in its own produce request. That capped the sink at roughly 400 msg/s in our setup, and we found that large batches could not be processed before their Kafka transactions timed out.

What

Stop setting the three options, so librdkafka's defaults apply, and producer_options can tune them. commit_transaction still flushes before the transaction ends, and recovery reads only the last committed key per partition, so neither depends on request boundaries.

It's also worth noting that, while not covered in this PR, we observed that there's no abort and retry behaviour when a write via a Kafka transaction times out in exactly-once ft mode.

Checklist

In my opinion, passing the existing tests is sufficient here, but happy to add tests if needed.

  • Unit tests added/updated
  • Integration tests added/updated
  • Documentation updated
  • Changelog updated

Breaking Changes?

  • Adapters (including configuration)

Describe Incompatible Changes

Default producer batching changes, and batch.size, batch.num.messages and linger.ms are no longer rejected when set in producer_options.

🤖 Code generated with Claude Code

@lalithsuresh
lalithsuresh requested a review from blp October 8, 2026 01:38
@jacksehr
jacksehr marked this pull request as ready for review October 8, 2026 02:56
@blp
blp enabled auto-merge October 12, 2026 22:52
auto-merge was automatically disabled October 22, 2026 23:34

Head branch was pushed to by a user without write access

@mihaibudiu

Copy link
Copy Markdown
Contributor

Can you please rebase again?

@jacksehr
jacksehr force-pushed the ft-kafka-producer-batching branch from 7a46727 to 60aa0e7 Compare October 30, 2026 07:42
jacksehr and others added 3 commits November 2, 2026 14:15
…put producer.

The producer forced batch.size, batch.num.messages and linger.ms to their
minimums, so every message went in its own request and capped the sink at
about 400 messages/s.  Leave the three settings at the librdkafka defaults.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: Jack She <jackshe2702@gmail.com>
Add a changelog entry and a note in the Kafka sink docs that batching is left
to librdkafka and can be tuned in fault_tolerance.producer_options.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: Jack She <jackshe2702@gmail.com>
@jacksehr
jacksehr force-pushed the ft-kafka-producer-batching branch from 60aa0e7 to 33466dd Compare November 2, 2026 03:22
@mihaibudiu
mihaibudiu added this pull request to the merge queue Nov 2, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Nov 2, 2026
@mihaibudiu
mihaibudiu added this pull request to the merge queue Nov 2, 2026
Merged via the queue into feldera:main with commit 638f7de Nov 3, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants