← 输入框|github.com|刷新
Skip to content

[connectors] Kafka: fix flaky check of output memory after an outage. - #7789

Merged
blp merged 1 commit into
mainfrom
kafka-ft-stale-stats
Nov 3, 2026
Merged

blp merged 1 commit into
mainfrom
kafka-ft-stale-stats

Conversation

@blp

@blp blp commented Nov 2, 2026

Copy link
Copy Markdown
Member

an_outage_aborts_and_replays_the_transaction failed in the merge queue for #7778 (run):

assertion `left == right` failed: the spool must be released at commit
  left: 190
 right: 0

Cause

KafkaOutputEndpoint::memory() is the spool plus the librdkafka queue bytes. The queue part updates only in the stats callback (statistics.interval.ms, 10 s by default). The spool was released. But the last callback ran during the outage, when 5 records (5 × 38 = 190 bytes) were queued, and no callback ran between the commit and the assertion.

Fix

The test now gets statistics every 100 ms and waits up to 5 s for memory() to reach 0.

Validation

Change Result
Old assertion, stale 190 bytes put in the memory reporter before the check fails with left: 190 (same as CI)
New wait, same stale value passes
New wait, clear_spool() skipped in batch_end fails (still catches a real leak)
All mock-cluster tests in kafka::ft::output::tests 12 passed

/// A connected sink on `cluster` with a one second transaction timeout, so
/// that a delivery failure turns into an abortable transaction quickly.
/// that a delivery failure turns into an abortable transaction quickly, and
/// statistics every 100 ms, so that `memory()` soon shows the producer queue.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

soon?

@blp blp Nov 2, 2026 •

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, "soon" was vague. In b242a6e the comment now gives the bound: with a 100 ms statistics interval, the producer queue part of memory() is at most about 100 ms old (the default is 10 s). The test waits up to 5 s for memory() to reach 0.

(Claude, on behalf of Ben)

@mythical-fred-oss

Copy link
Copy Markdown
Contributor

LGTM

What I ran

Check Result
cargo test --locked -p dbsp_connectors --lib kafka::ft::output::tests 12 mock-cluster tests passed. The 5 real::* tests failed because this sandbox has no Kafka broker, which is unrelated to this PR.
an_outage_aborts_and_replays_the_transaction run 8 times in a row 8/8 passed (about 9–13 s each)
rustfmt --check on the changed file clean

Fix check: memory() takes the queue size from the stats callback. The producer is a ThreadedProducer, so that callback keeps firing in the background while the test waits. Setting a 100 ms interval with a 5 s wait leaves a wide margin, and the fix is not timing-sensitive in a flaky way. Checking kafka_output_replay_spool_bytes right after the wait still catches a real spool leak.

Feldera PR gates: all pass. This changes only a test, so the docs, SQL type coverage and breaking-change gates don't apply.

`an_outage_aborts_and_replays_the_transaction` checked that
`KafkaOutputEndpoint::memory()` is 0 right after the commit. `memory()`
also counts the bytes in the librdkafka queue, which updates only in the
statistics callback (every 10 s by default). When the last callback ran
during the outage, `memory()` still reported the 5 queued records
(190 bytes) after the commit, and the test failed in the merge queue.

The test now gets statistics every 100 ms and waits for `memory()` to
reach 0.

Signed-off-by: Ben Pfaff <blp@feldera.com>
@blp
blp force-pushed the kafka-ft-stale-stats branch from 0b7b83a to b242a6e Compare November 2, 2026 19:20
@blp blp self-assigned this Nov 2, 2026
@blp blp added connectors Issues related to the adapters/connectors crate low priority Should not be prioritized for at least two subsequent sprints QA Testing and quality assurance rust Pull requests that update Rust code labels Nov 2, 2026
@blp
blp enabled auto-merge November 2, 2026 19:23
@blp
blp disabled auto-merge November 2, 2026 20:38
@blp
blp added this pull request to the merge queue Nov 2, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Nov 3, 2026
@mythical-fred-oss

Copy link
Copy Markdown
Contributor

Verdict: pre-existing flaky test on main, not caused by this PR. High confidence.

Failed job Adapter Tests / Test Adapters
Failing test transport::kafka::ft::output::tests::real::restart_skips_a_commit_whose_reply_was_lost
Panic log line 1657: the consumer must reach the high watermark: () (60s timeout reading the committed topic)

This PR's diff only touches the mock-cluster endpoint() helper and the an_outage_aborts_and_replays_the_transaction assertion in tests.rs. The failing test lives in real.rs with its own separate endpoint(), so it isn't exercised by this change.

Already tracked as a known flake in #7765 (timing race between the test's held-commit/read-wait budget and the broker's own 60s transaction.timeout.ms) — this run is a repeat occurrence, now noted there.

Next step: re-queueing should pass; this test doesn't fail deterministically, just intermittently under this timing race.

@mihaibudiu
mihaibudiu added this pull request to the merge queue Nov 3, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Nov 3, 2026
@mythical-fred-oss

Copy link
Copy Markdown
Contributor

Verdict: pre-existing flaky test on main, not caused by this PR. High confidence.

Failed job Unit Tests / Rust Unit Tests (aarch64)
Failing test operator::dynamic::sharded_accumulator::tests::take_received
Panic log line 3360: receiving exchange messages ...: UnknownExchange(0) inside dbsp's multihost test harness

This PR's diff only touches crates/connectors/src/transport/kafka/ft/output/tests.rs. The failing test lives in dbsp's sharded-accumulator/multihost harness, which this PR never touches — it's a race in the TCP exchange registration between hosts, in code under active rework from #7745 (merged 2026-11-02).

Filed as a new issue: #7800 (no prior report of this exact failure).

Next step: re-queueing should pass; this looks like an intermittent timing race in the multihost test harness, not a deterministic failure.

@blp
blp added this pull request to the merge queue Nov 3, 2026
Merged via the queue into main with commit a59c92d Nov 3, 2026
1 check passed
@blp
blp deleted the kafka-ft-stale-stats branch November 3, 2026 20:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

connectors Issues related to the adapters/connectors crate low priority Should not be prioritized for at least two subsequent sprints QA Testing and quality assurance rust Pull requests that update Rust code

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants