← 输入框|github.com|刷新
Skip to content

[connectors] Kafka: fix input skip and at-least-once output loss found by Antithesis - #7861

Merged
blp merged 5 commits into
mainfrom
kafka-speedrun-fixes
Nov 10, 2026
Merged

blp merged 5 commits into
mainfrom
kafka-speedrun-fixes

Conversation

@blp

@blp blp commented Nov 5, 2026 •

Copy link
Copy Markdown
Member

An Antithesis report on v0.363.0 found six Kafka problems. Three were fixed on main since then: the replay shift (9c4b63e) and the failed and timed-out exactly-once transactions (6ad915e). This PR fixes two more, plus a panic left over from the third, and hardens the replay check that missed the replay shift.

Commit Problem Fix
Kafka input: don't resume past messages drained at start-up A stop during connector start-up can checkpoint a resume offset past messages drained from the main consumer queue but not yet read. The restart skips them. Seen with at-least-once (18 records lost). It also explains the report's one unexplained exactly-once table mismatch: in run 22609803, the exactly-once pipeline resumed from a checkpoint, was stopped during the replay that followed, and checkpointed offsets 478 records past the records it had processed. After a resume, the drain can hold a whole backlog. Staged ranges start at the first unread message, not next_offset.
Kafka output: wait for delivery at the end of each batch At-least-once Kafka output had no batch_end, so a checkpoint could cover records still queued in librdkafka. A crash after the checkpoint lost them. batch_end waits until every record sent so far is acknowledged or failed. Shutdown ends the wait.
Kafka exactly-once output: don't panic after a failed batch_start A batch_start that shutdown cancels leaves no batch open, and push_key/batch_end then hit unreachable!. They return errors instead.
Fail replay that feeds a step before the previous commit The per-transaction replay check compares what the endpoints read, not the transaction their input entered, so the replay shift fixed in 9c4b63e lost exactly-once output without any error. Replay fails with ReplayFailure if it reaches a transaction boundary with the next step already handed to the endpoints. It checks before it starts the commit.

Not addressed: a pipeline that restarts while a broker is unreachable still fails initialization for good. The single-shot initialization calls are a deliberate choice, so that needs a separate decision.

For review: the batch_end wait adds a broker round trip to each at-least-once output batch. Output buffering spreads it over several steps.

Testing

  • Each new test fails with its fix reverted and passes with it.
  • cargo test -p dbsp_connectors --lib -- transport::kafka: 82 passed.
  • cargo test -p static-connectors --test connector_suites -- kafka: 33 passed, 1 ignored, as on main.
  • cargo test -p static-connectors --test connector_suites -- controller:: server::: 144 passed.
  • cargo clippy -p dbsp_connectors -p static-connectors --all-targets: no warnings.

@blp
blp requested a review from bestouff November 5, 2026 19:39
.unwrap()
.0;
}
Ok(())

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bug (medium): this returns Ok(()) once pending is empty, even when deliveries failed. If the broker stays down past message.timeout.ms (300 s by default), librdkafka fails the records, complete_delivery removes them, and the batch still ends cleanly. A checkpoint can then cover records that never arrived, which is the same loss this PR fixes, just after a longer outage. One option: track failures per batch, for example a failure count since the last batch_end, and return Err here. Note that the controller (outprobe_batch_end) treats that error as non-fatal, so it may also need to keep the step out of the checkpoint. At minimum, please mention the remaining gap in the doc comment and the commit message.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One correction: for at-least-once output the default message.timeout.ms is 0 (unlimited, see crates/feldera-types/src/transport/kafka.rs), not 300 s. So by default a record never fails because of a timeout. Records are lost only with a finite timeout that the user sets, or when Kafka refuses a record (for example, if it is too large), and both cases are reported as errors. If batch_end returned Err here, nothing would change, because the controller treats batch_end errors as non-fatal and the checkpoint still covers the step. To fix this, the controller would have to change, which is out of scope for this PR. e00198f documents the gap in the batch_end doc comment and in the Kafka sink docs, and adds delivery_failure_ends_batch_end.

DELAY_SPLIT.lock().unwrap().remove(topic);
assert!(
DRAINED.lock().unwrap().get(topic).copied().unwrap_or(0) > 0,
"no message reached the main queue, so this test checked nothing"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pitfall (medium, flaky test): whether this passes depends on librdkafka fetching into the main queue during the fixed 2 s DELAY_SPLIT sleep. On a loaded CI runner the fetch can come late, DRAINED stays at 0, and the test panics. Two ways to make it deterministic: have the reader wait until the main queue holds at least one message (with a generous upper bound), or make this case skip or retry instead of fail. Otherwise this test will turn red at random.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in e00198f: the test now recreates the reader (up to 10 tries) until at least one message is drained from the main queue, so a slow fetch no longer fails it. With the fix reverted, the test still fails.

@mythical-fred-oss

Copy link
Copy Markdown
Contributor

What I ran

Command Result
cargo test --locked -p dbsp_connectors --lib -- kafka::nonft::output kafka::ft::output::tests::output_after 9 passed, including batch_end_waits_for_delivery, shutdown_ends_batch_end and output_after_a_failed_batch_start_is_an_error
test_input_queue_before_drained_messages Could not run: there is no Kafka broker in this environment

Gates

  • Flaky tests (medium): the start-up drain test depends on a fixed 2 s sleep (see the inline comment). This is the only gate I'd block on until it is fixed.
  • SQL type coverage: not applicable. The changes are to transport-level delivery and offsets, not encoding.
  • Docs: not applicable.

Missing tests (medium)

  1. Delivery fails while batch_end is waiting (for example message.timeout.ms=1000 with the broker down). Today batch_end returns Ok; see the inline comment.
  2. batch_end with nothing pending (no records sent) returns at once.
  3. Start-up drain combined with the header filter, where the first drained offset is filtered out: the staged range should start at the first admitted unread offset, and replay should stay deterministic.
  4. The same drain scenario with exactly-once (the PR says it is affected too, but only at-least-once is tested).

Pitfall (medium): with the broker down, batch_end now blocks the output thread for up to message.timeout.ms per batch. That is intended, but the connector docs or the PR description should mention it, because it shows up as output backpressure.

@blp

blp commented Nov 5, 2026

Copy link
Copy Markdown
Member Author

Summary review, addressed in e00198f:

  • Added tests: delivery_failure_ends_batch_end (finite timeout) and batch_end_without_pending_records.
  • Documented the blocking and backpressure while the broker is down in the batch_end doc comment and in docs.feldera.com/docs/connectors/sinks/kafka.md.
  • Exactly-once drain: test_input_queue_before_drained_messages already runs with exactly-once, because DummyInputConsumer::pipeline_fault_tolerance returns ExactlyOnce.
  • Drain with a header filter: not added. Messages that the filter drops never go into the unread map, so the first unread offset does not change.

@blp
blp force-pushed the kafka-speedrun-fixes branch from e00198f to 20c5ed5 Compare November 5, 2026 23:17

@bestouff bestouff left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A small nit (see inline comment), and maybe squash commit 5.

break (endpoint, receiver, reader);
}
assert!(
attempts < 10,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

<= if you want to match the printf

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that it's correct as is, because the first iteration increments attempts to 1, ..., and the tenth iteration increments it to 10, so that the message will complain about attempt 10.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah that's my fault, sorry about that; I'm just so used to start counting at 0 ...

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This would be better written as a for loop that would make it obvious how we're counting:

for attempt in 1..

except that then we wouldn't be able to break with a value, which is useful in this case. I've had that annoyance multiple times. But I don't have a better language design.

@bestouff

bestouff commented Nov 9, 2026

Copy link
Copy Markdown
Contributor

Oops, Claude found another one:

Shutdown now produces endpoint errors where it did not before. The nonft batch_end bails when shutdown cancels the wait. The controller wrapper at crates/connectors/src/controller.rs:10718-10727 passes that straight to output_transport_error, which logs at ERROR and bumps the endpoint's error count. The encoder path right above it at controller.rs:9262-9270 deliberately suppresses errors when the shutdown token is cancelled, with a comment explaining why. The exactly-once case is worse: after a shutdown-cancelled batch_start the controller keeps encoding, so every push_buffer or push_key of that batch now reports one error each. Suggest either returning Ok(()) at shutdown in the endpoints, or adding the same shutdown_token().is_cancelled() check to the OutputProbe::batch_end, push_buffer and push_key wrappers.

@blp

blp commented Nov 9, 2026

Copy link
Copy Markdown
Member Author

@bestouff Thanks for the reviews, I'll fix it up and merge.

blp added 4 commits November 9, 2026 10:42
…-up.

At start-up, the fault-tolerant Kafka reader drains the main consumer
queue, which can hold messages that librdkafka fetched before the
partition queues were split.  Draining advances each partition's
`next_offset`, but the messages wait unread until the reader runs.  The
staged offset ranges started at `next_offset`, so a step queued before
the reader read them, as when the pipeline stops during start-up,
recorded an empty range past them, and a restart from that step skipped
them for good.  Antithesis found an at-least-once pipeline that lost 18
records this way.  Exactly-once pipelines have the same problem: in
another run, an exactly-once pipeline resumed from a checkpoint, was
stopped during the replay that followed, and checkpointed offsets 478
records past the records it had processed.  After a resume, the drain
can hold a whole backlog.

Now the ranges start at the first unread message.

The test uses a test-only delay between assigning partitions and
splitting their queues, so that messages reach the main queue, and
retries until they do.

Signed-off-by: Ben Pfaff <blp@feldera.com>
The at-least-once Kafka output had no `batch_end`, so the controller
counted a batch as transmitted as soon as librdkafka queued its records.
A checkpoint could then cover records that were still queued, for
example while a broker was unreachable, and a crash after that
checkpoint lost them: the pipeline resumed past them and never sent
them again.  Antithesis found this.

Now `batch_end` waits until Kafka acknowledges or fails every record
sent so far.  Shutdown ends the wait.  This adds a broker round trip to
each output batch; output buffering amortizes it over several steps.

While the broker is unreachable, `batch_end` blocks the output thread,
so output backs up as it does for any slow sink.  A record that fails,
because Kafka refuses it or because a finite `message.timeout.ms`
expires, is still lost even if a checkpoint covers it; the default
timeout is unlimited.  The Kafka sink docs now say so.

Signed-off-by: Ben Pfaff <blp@feldera.com>
…tch_start.

`batch_start` fails only when shutdown cancels it, and then it leaves no
batch open.  The controller keeps encoding the batch anyway, so
`push_key` and `batch_end` hit `unreachable!`.  Now they return errors.
The transaction never commits, so a restart sends its records again.

Signed-off-by: Ben Pfaff <blp@feldera.com>
Replay checks that each replayed transaction re-ingests the input
recorded for it.  The check compares what the input endpoints read, not
the transaction that their input entered, so it missed a bug, fixed in
9c4b63e, that handed a journaled step to the endpoints before the
previous transaction committed.  The commit then took the next
transaction's input, and exactly-once Kafka output skipped that
transaction as already written, so its changes were lost without any
error.

Now replay fails with `ReplayFailure` if it reaches a transaction
boundary with the next step already handed to the endpoints.  It checks
before it starts the commit, so it stops the pipeline instead of
committing the wrong input.

The test restores the old eager feeding through a test-only switch and
expects the failure.  Without the check, replay finishes with no error.

Signed-off-by: Ben Pfaff <blp@feldera.com>
@blp

blp commented Nov 9, 2026 •

Copy link
Copy Markdown
Member Author

Claude here, replying for Ben.

@bestouff Thanks, the shutdown finding was right. Fixed in 89af86c: OutputProbe now sends batch_start, push_buffer, push_key and batch_end errors through one helper, endpoint_error. While the shutdown token is cancelled, the helper logs the error at debug and does not report it. This is the same as what push_batch_to_encoder does for encoder errors. I fixed it in the controller, not in each endpoint, so other sinks that a shutdown cuts short get the same behavior.

I also squashed the "address review" commit into the commits it fixed, as you suggested.

@blp
blp force-pushed the kafka-speedrun-fixes branch from 20c5ed5 to 594140d Compare November 9, 2026 18:42
A shutdown makes the Kafka sink's `batch_end` give up its wait for
delivery, and makes an exactly-once `batch_start` fail, after which each
`push_buffer` and `push_key` of the batch fails too.  `OutputProbe`
reported all of these as endpoint errors, which logs at ERROR and
increments the endpoint's error count on every stop of a busy sink.

`OutputProbe` now skips the report while the pipeline shuts down, as
`push_batch_to_encoder` already does for encoder errors.

The new test has an endpoint whose `batch_end` fails after a stop
begins, and checks that the error does not reach the error callback.

Signed-off-by: Ben Pfaff <blp@feldera.com>
@blp
blp force-pushed the kafka-speedrun-fixes branch from 83132e8 to 89af86c Compare November 9, 2026 18:59
@blp
blp added this pull request to the merge queue Nov 9, 2026
Merged via the queue into main with commit 900c255 Nov 10, 2026
1 check passed
@blp
blp deleted the kafka-speedrun-fixes branch November 10, 2026 00:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants