← 输入框|github.com|刷新
Skip to content

[adapters] Support caller-supplied keys in exactly-once Kafka output - #7433

Merged
mihaibudiu merged 1 commit into
feldera:mainfrom
jacksehr:7355-ft-kafka-keyed-output-split
Nov 3, 2026
Merged

mihaibudiu merged 1 commit into
feldera:mainfrom
jacksehr:7355-ft-kafka-keyed-output-split

Conversation

@jacksehr

@jacksehr jacksehr commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

Describe Manual Test Plan

Deployed to our cluster, where we ingested 10M+ records with no issues. Worth noting it was patched into the enterprise image alongside #7432, which should be orthogonal.

It was also validated locally in Docker, with a JSON debezium sink using key_fields under exactly_once: every keyed record was delivered, no key spanned partitions, and there were no duplicates after a graceful restart or a force-killed pipeline pod.

Why

With exactly-once fault tolerance, the Kafka output stored its dedup position as the message key. A format that supplies its own key (JSON debezium, or key_fields) had nowhere to put it, so the connector rejected those messages. Keyless messages went round-robin over partitions, so nothing was ordered per key.

In order to maintain ordering within a key (i.e. a tenant id), while still benefitting from spreading consumer work over partitions, we need to be able to supply message keys while still persisting the exactly_once dedup positions. Fixes #7355.

What

  • When the caller supplies a key, it becomes the message key and the position moves to a __feldera_position header, written before any other header. librdkafka partitions by key hash, so records with one key stay in order in one partition.
  • Keyless messages are unchanged: the position stays the key, which survives compaction.
  • Keyed tombstones and keyed messages over message.max.bytes are refused before they take a substep. Compaction eventually removes a tombstone, and recovery would then read an older position and rewrite committed output. Keyed messages with a value are safe on compacted topics.
  • Docs and changelog describe the keyed format and the reserved header.

Checklist

  • Unit tests added/updated
  • Integration tests added/updated
  • Documentation updated
  • Changelog updated

Breaking Changes?

  • Adapters (including configuration)

Describe Incompatible Changes

Keyed output with exactly-once fault tolerance now works instead of erroring. Formats that write deletions as tombstones (Avro confluent_jdbc, JSON redis) still error in this mode.

🤖 Generated with Claude Code

@blp

blp commented Oct 12, 2026

Copy link
Copy Markdown
Member

@jacksehr This is submitted as a draft PR. Are you ready for me to have a look at it?

@jacksehr

Copy link
Copy Markdown
Contributor Author

Ah yep it should be good to go, I'll resolve conflicts and mark it ready in a sec.

@jacksehr
jacksehr marked this pull request as ready for review October 12, 2026 22:28
@blp

blp commented Oct 13, 2026 •

Copy link
Copy Markdown
Member

Is it safe to use user-supplied keys with exactly once fault tolerance? If the Kafka compactor deletes messages, then this would break replay.

EDIT: Oh I see, we generate unique keys.

@blp blp left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm pretty happy with this but I'd like to see it all squashed into a single commit, unless you see some reason to keep one or two commits separate.

Thank you!

"Kafka output transport in exactly once fault-tolerant mode does not support messages with a key but no value (tombstones), which the 'confluent_jdbc' and 'redis' formats produce for deletions. Use a format that represents a deletion with a value, such as 'debezium', or at-least-once fault tolerance."
);
};
let size = key.len() + val.len() + POSITION_HEADER.len() + position_json.len();

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't know how Kafka measures message sizes but I guess this should include the headers as well.

@jacksehr
jacksehr force-pushed the 7355-ft-kafka-keyed-output-split branch from 2cb1529 to 6fad954 Compare October 14, 2026 02:22
@blp
blp enabled auto-merge October 20, 2026 16:52
auto-merge was automatically disabled October 30, 2026 07:59

Head branch was pushed to by a user without write access

@jacksehr
jacksehr force-pushed the 7355-ft-kafka-keyed-output-split branch from a77fa47 to 57d8e22 Compare October 30, 2026 07:59
@mihaibudiu

Copy link
Copy Markdown
Contributor

sorry, there's one more conflict to resolve

@jacksehr
jacksehr force-pushed the 7355-ft-kafka-keyed-output-split branch from 57d8e22 to b013d83 Compare November 2, 2026 03:22
@mihaibudiu

Copy link
Copy Markdown
Contributor

This PR does not seem to compile as it is, I haven't checked to see if some code changed underneath it.

@mihaibudiu
mihaibudiu added this pull request to the merge queue Nov 3, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Nov 3, 2026
@mihaibudiu

Copy link
Copy Markdown
Contributor

@jacksehr the tests you wrote are failing in CI.

With exactly-once fault tolerance, the Kafka output stored its dedup
position as the message key, so formats that supply their own key (JSON
`debezium`, or `key_fields`) were rejected, and keyless messages went
round-robin over partitions with no per-key ordering.

When the caller supplies a key, it becomes the message key and the position
moves to a `__feldera_position` header, written before any other header.
librdkafka partitions by key hash, so records with one key stay in order in
one partition.  Keyless messages are unchanged: the position stays the key.

Recovery reads the last message of each partition, which compaction always
keeps for its key.  Keyed tombstones are refused before they take a
substep, since compaction eventually removes them.  Keyed messages over
`message.max.bytes`, counting the position header and every configured and
per-message header, are refused the same way.

Fixes feldera#7355.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Qnp9Jkrw6hY6VeJSzziMHv
@jacksehr
jacksehr force-pushed the 7355-ft-kafka-keyed-output-split branch from c4fb366 to 071f5a7 Compare November 3, 2026 06:04
@mihaibudiu
mihaibudiu added this pull request to the merge queue Nov 3, 2026
Merged via the queue into feldera:main with commit 123be56 Nov 3, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Keyed Kafka output for exactly_once fault tolerance

3 participants