Repository navigation
[connectors] Kafka: fix input skip and at-least-once output loss found by Antithesis - #7861
Conversation
| .unwrap() | ||
| .0; | ||
| } | ||
| Ok(()) |
There was a problem hiding this comment.
bug (medium): this returns Ok(()) once pending is empty, even when deliveries failed. If the broker stays down past message.timeout.ms (300 s by default), librdkafka fails the records, complete_delivery removes them, and the batch still ends cleanly. A checkpoint can then cover records that never arrived, which is the same loss this PR fixes, just after a longer outage. One option: track failures per batch, for example a failure count since the last batch_end, and return Err here. Note that the controller (outprobe_batch_end) treats that error as non-fatal, so it may also need to keep the step out of the checkpoint. At minimum, please mention the remaining gap in the doc comment and the commit message.
There was a problem hiding this comment.
One correction: for at-least-once output the default message.timeout.ms is 0 (unlimited, see crates/feldera-types/src/transport/kafka.rs), not 300 s. So by default a record never fails because of a timeout. Records are lost only with a finite timeout that the user sets, or when Kafka refuses a record (for example, if it is too large), and both cases are reported as errors. If batch_end returned Err here, nothing would change, because the controller treats batch_end errors as non-fatal and the checkpoint still covers the step. To fix this, the controller would have to change, which is out of scope for this PR. e00198f documents the gap in the batch_end doc comment and in the Kafka sink docs, and adds delivery_failure_ends_batch_end.
| DELAY_SPLIT.lock().unwrap().remove(topic); | ||
| assert!( | ||
| DRAINED.lock().unwrap().get(topic).copied().unwrap_or(0) > 0, | ||
| "no message reached the main queue, so this test checked nothing" |
There was a problem hiding this comment.
pitfall (medium, flaky test): whether this passes depends on librdkafka fetching into the main queue during the fixed 2 s DELAY_SPLIT sleep. On a loaded CI runner the fetch can come late, DRAINED stays at 0, and the test panics. Two ways to make it deterministic: have the reader wait until the main queue holds at least one message (with a generous upper bound), or make this case skip or retry instead of fail. Otherwise this test will turn red at random.
There was a problem hiding this comment.
Fixed in e00198f: the test now recreates the reader (up to 10 tries) until at least one message is drained from the main queue, so a slow fetch no longer fails it. With the fix reverted, the test still fails.
|
What I ran
Gates
Missing tests (medium)
Pitfall (medium): with the broker down, |
|
Summary review, addressed in e00198f:
|
e00198f to
20c5ed5
Compare
bestouff
left a comment
There was a problem hiding this comment.
A small nit (see inline comment), and maybe squash commit 5.
| break (endpoint, receiver, reader); | ||
| } | ||
| assert!( | ||
| attempts < 10, |
There was a problem hiding this comment.
<= if you want to match the printf
There was a problem hiding this comment.
I think that it's correct as is, because the first iteration increments attempts to 1, ..., and the tenth iteration increments it to 10, so that the message will complain about attempt 10.
There was a problem hiding this comment.
Yeah that's my fault, sorry about that; I'm just so used to start counting at 0 ...
There was a problem hiding this comment.
This would be better written as a for loop that would make it obvious how we're counting:
for attempt in 1..
except that then we wouldn't be able to break with a value, which is useful in this case. I've had that annoyance multiple times. But I don't have a better language design.
|
Oops, Claude found another one:
|
|
@bestouff Thanks for the reviews, I'll fix it up and merge. |
…-up. At start-up, the fault-tolerant Kafka reader drains the main consumer queue, which can hold messages that librdkafka fetched before the partition queues were split. Draining advances each partition's `next_offset`, but the messages wait unread until the reader runs. The staged offset ranges started at `next_offset`, so a step queued before the reader read them, as when the pipeline stops during start-up, recorded an empty range past them, and a restart from that step skipped them for good. Antithesis found an at-least-once pipeline that lost 18 records this way. Exactly-once pipelines have the same problem: in another run, an exactly-once pipeline resumed from a checkpoint, was stopped during the replay that followed, and checkpointed offsets 478 records past the records it had processed. After a resume, the drain can hold a whole backlog. Now the ranges start at the first unread message. The test uses a test-only delay between assigning partitions and splitting their queues, so that messages reach the main queue, and retries until they do. Signed-off-by: Ben Pfaff <blp@feldera.com>
The at-least-once Kafka output had no `batch_end`, so the controller counted a batch as transmitted as soon as librdkafka queued its records. A checkpoint could then cover records that were still queued, for example while a broker was unreachable, and a crash after that checkpoint lost them: the pipeline resumed past them and never sent them again. Antithesis found this. Now `batch_end` waits until Kafka acknowledges or fails every record sent so far. Shutdown ends the wait. This adds a broker round trip to each output batch; output buffering amortizes it over several steps. While the broker is unreachable, `batch_end` blocks the output thread, so output backs up as it does for any slow sink. A record that fails, because Kafka refuses it or because a finite `message.timeout.ms` expires, is still lost even if a checkpoint covers it; the default timeout is unlimited. The Kafka sink docs now say so. Signed-off-by: Ben Pfaff <blp@feldera.com>
…tch_start. `batch_start` fails only when shutdown cancels it, and then it leaves no batch open. The controller keeps encoding the batch anyway, so `push_key` and `batch_end` hit `unreachable!`. Now they return errors. The transaction never commits, so a restart sends its records again. Signed-off-by: Ben Pfaff <blp@feldera.com>
Replay checks that each replayed transaction re-ingests the input recorded for it. The check compares what the input endpoints read, not the transaction that their input entered, so it missed a bug, fixed in 9c4b63e, that handed a journaled step to the endpoints before the previous transaction committed. The commit then took the next transaction's input, and exactly-once Kafka output skipped that transaction as already written, so its changes were lost without any error. Now replay fails with `ReplayFailure` if it reaches a transaction boundary with the next step already handed to the endpoints. It checks before it starts the commit, so it stops the pipeline instead of committing the wrong input. The test restores the old eager feeding through a test-only switch and expects the failure. Without the check, replay finishes with no error. Signed-off-by: Ben Pfaff <blp@feldera.com>
|
Claude here, replying for Ben. @bestouff Thanks, the shutdown finding was right. Fixed in 89af86c: I also squashed the "address review" commit into the commits it fixed, as you suggested. |
20c5ed5 to
594140d
Compare
A shutdown makes the Kafka sink's `batch_end` give up its wait for delivery, and makes an exactly-once `batch_start` fail, after which each `push_buffer` and `push_key` of the batch fails too. `OutputProbe` reported all of these as endpoint errors, which logs at ERROR and increments the endpoint's error count on every stop of a busy sink. `OutputProbe` now skips the report while the pipeline shuts down, as `push_batch_to_encoder` already does for encoder errors. The new test has an endpoint whose `batch_end` fails after a stop begins, and checks that the error does not reach the error callback. Signed-off-by: Ben Pfaff <blp@feldera.com>
83132e8 to
89af86c
Compare
An Antithesis report on v0.363.0 found six Kafka problems. Three were fixed on main since then: the replay shift (9c4b63e) and the failed and timed-out exactly-once transactions (6ad915e). This PR fixes two more, plus a panic left over from the third, and hardens the replay check that missed the replay shift.
next_offset.batch_end, so a checkpoint could cover records still queued in librdkafka. A crash after the checkpoint lost them.batch_endwaits until every record sent so far is acknowledged or failed. Shutdown ends the wait.batch_startthat shutdown cancels leaves no batch open, andpush_key/batch_endthen hitunreachable!.ReplayFailureif it reaches a transaction boundary with the next step already handed to the endpoints. It checks before it starts the commit.Not addressed: a pipeline that restarts while a broker is unreachable still fails initialization for good. The single-shot initialization calls are a deliberate choice, so that needs a separate decision.
For review: the
batch_endwait adds a broker round trip to each at-least-once output batch. Output buffering spreads it over several steps.Testing
cargo test -p dbsp_connectors --lib -- transport::kafka: 82 passed.cargo test -p static-connectors --test connector_suites -- kafka: 33 passed, 1 ignored, as on main.cargo test -p static-connectors --test connector_suites -- controller:: server::: 144 passed.cargo clippy -p dbsp_connectors -p static-connectors --all-targets: no warnings.