Repository navigation
[adapters] Support caller-supplied keys in exactly-once Kafka output - #7433
Conversation
|
@jacksehr This is submitted as a draft PR. Are you ready for me to have a look at it? |
|
Ah yep it should be good to go, I'll resolve conflicts and mark it ready in a sec. |
|
Is it safe to use user-supplied keys with exactly once fault tolerance? If the Kafka compactor deletes messages, then this would break replay. EDIT: Oh I see, we generate unique keys. |
blp
left a comment
There was a problem hiding this comment.
I'm pretty happy with this but I'd like to see it all squashed into a single commit, unless you see some reason to keep one or two commits separate.
Thank you!
| "Kafka output transport in exactly once fault-tolerant mode does not support messages with a key but no value (tombstones), which the 'confluent_jdbc' and 'redis' formats produce for deletions. Use a format that represents a deletion with a value, such as 'debezium', or at-least-once fault tolerance." | ||
| ); | ||
| }; | ||
| let size = key.len() + val.len() + POSITION_HEADER.len() + position_json.len(); |
There was a problem hiding this comment.
I don't know how Kafka measures message sizes but I guess this should include the headers as well.
2cb1529 to
6fad954
Compare
Head branch was pushed to by a user without write access
a77fa47 to
57d8e22
Compare
|
sorry, there's one more conflict to resolve |
57d8e22 to
b013d83
Compare
|
This PR does not seem to compile as it is, I haven't checked to see if some code changed underneath it. |
|
@jacksehr the tests you wrote are failing in CI. |
With exactly-once fault tolerance, the Kafka output stored its dedup position as the message key, so formats that supply their own key (JSON `debezium`, or `key_fields`) were rejected, and keyless messages went round-robin over partitions with no per-key ordering. When the caller supplies a key, it becomes the message key and the position moves to a `__feldera_position` header, written before any other header. librdkafka partitions by key hash, so records with one key stay in order in one partition. Keyless messages are unchanged: the position stays the key. Recovery reads the last message of each partition, which compaction always keeps for its key. Keyed tombstones are refused before they take a substep, since compaction eventually removes them. Keyed messages over `message.max.bytes`, counting the position header and every configured and per-message header, are refused the same way. Fixes feldera#7355. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Qnp9Jkrw6hY6VeJSzziMHv
c4fb366 to
071f5a7
Compare
Describe Manual Test Plan
Deployed to our cluster, where we ingested 10M+ records with no issues. Worth noting it was patched into the enterprise image alongside #7432, which should be orthogonal.
It was also validated locally in Docker, with a JSON
debeziumsink usingkey_fieldsunderexactly_once: every keyed record was delivered, no key spanned partitions, and there were no duplicates after a graceful restart or a force-killed pipeline pod.Why
With exactly-once fault tolerance, the Kafka output stored its dedup position as the message key. A format that supplies its own key (JSON
debezium, orkey_fields) had nowhere to put it, so the connector rejected those messages. Keyless messages went round-robin over partitions, so nothing was ordered per key.In order to maintain ordering within a key (i.e. a tenant id), while still benefitting from spreading consumer work over partitions, we need to be able to supply message keys while still persisting the
exactly_oncededup positions. Fixes #7355.What
__feldera_positionheader, written before any other header. librdkafka partitions by key hash, so records with one key stay in order in one partition.message.max.bytesare refused before they take a substep. Compaction eventually removes a tombstone, and recovery would then read an older position and rewrite committed output. Keyed messages with a value are safe on compacted topics.Checklist
Breaking Changes?
Describe Incompatible Changes
Keyed output with exactly-once fault tolerance now works instead of erroring. Formats that write deletions as tombstones (Avro
confluent_jdbc, JSONredis) still error in this mode.🤖 Generated with Claude Code