Repository navigation
[connectors] Kafka: fix flaky check of output memory after an outage. - #7789
Conversation
| /// A connected sink on `cluster` with a one second transaction timeout, so | ||
| /// that a delivery failure turns into an abortable transaction quickly. | ||
| /// that a delivery failure turns into an abortable transaction quickly, and | ||
| /// statistics every 100 ms, so that `memory()` soon shows the producer queue. |
There was a problem hiding this comment.
Agreed, "soon" was vague. In b242a6e the comment now gives the bound: with a 100 ms statistics interval, the producer queue part of memory() is at most about 100 ms old (the default is 10 s). The test waits up to 5 s for memory() to reach 0.
(Claude, on behalf of Ben)
|
LGTM What I ran
Fix check: Feldera PR gates: all pass. This changes only a test, so the docs, SQL type coverage and breaking-change gates don't apply. |
`an_outage_aborts_and_replays_the_transaction` checked that `KafkaOutputEndpoint::memory()` is 0 right after the commit. `memory()` also counts the bytes in the librdkafka queue, which updates only in the statistics callback (every 10 s by default). When the last callback ran during the outage, `memory()` still reported the 5 queued records (190 bytes) after the commit, and the test failed in the merge queue. The test now gets statistics every 100 ms and waits for `memory()` to reach 0. Signed-off-by: Ben Pfaff <blp@feldera.com>
0b7b83a to
b242a6e
Compare
|
Verdict: pre-existing flaky test on
This PR's diff only touches the mock-cluster Already tracked as a known flake in #7765 (timing race between the test's held-commit/read-wait budget and the broker's own 60s Next step: re-queueing should pass; this test doesn't fail deterministically, just intermittently under this timing race. |
|
Verdict: pre-existing flaky test on
This PR's diff only touches Filed as a new issue: #7800 (no prior report of this exact failure). Next step: re-queueing should pass; this looks like an intermittent timing race in the multihost test harness, not a deterministic failure. |
an_outage_aborts_and_replays_the_transactionfailed in the merge queue for #7778 (run):Cause
KafkaOutputEndpoint::memory()is the spool plus the librdkafka queue bytes. The queue part updates only in thestatscallback (statistics.interval.ms, 10 s by default). The spool was released. But the last callback ran during the outage, when 5 records (5 × 38 = 190 bytes) were queued, and no callback ran between the commit and the assertion.Fix
The test now gets statistics every 100 ms and waits up to 5 s for
memory()to reach 0.Validation
left: 190(same as CI)clear_spool()skipped inbatch_endkafka::ft::output::tests