← 输入框|github.com|刷新
Skip to content

[connectors] Unit test fix: Release the replay gates lock before calling a gate. - #7753

Merged
blp merged 1 commit into
mainfrom
replay-gate-lock
Nov 2, 2026
Merged

blp merged 1 commit into
mainfrom
replay-gate-lock

Conversation

@blp

@blp blp commented Oct 30, 2026 •

Copy link
Copy Markdown
Member

REPLAY_GATES hooks hold an HTTP input endpoint's replay in tests. The endpoint's task called its gate inside if let on the locked map, so the lock stayed held while the gate waited (in edition 2024, too, a temporary in an if let scrutinee lives through the then block), and every other HTTP endpoint's replay blocked on it.

replay_does_not_feed_http_input_during_commit gates the second of two endpoints until the first transaction commits. When the second endpoint's gate ran first, the first endpoint could not replay, so the step never finished, and the test failed after its timeout ("replay never finished"). That happened once in about 70 runs of the controller suite.

The task now takes the gate out of the map, releasing the lock, before it calls the gate. The gate also blocked its tokio worker thread, so with one worker (TOKIO_WORKER_THREADS=1, or a runner whose available_parallelism() is 1) no other endpoint could replay at all; it now waits in block_in_place. Test-only code; no production path changes.

Describe Manual Test Plan

New unit test replay_gate_does_not_block_other_endpoints: one endpoint's gate waits for a second endpoint to finish its replay. It fails every time with the old code ("the other endpoint could not replay while the gate waited") and passes with the fix, and it also fails with TOKIO_WORKER_THREADS=1 without block_in_place. cargo test -p dbsp_connectors --lib transport::http::input passes 7 of 7 with 1 and with 2 tokio workers, and clippy is clean.

Checklist

  • Unit tests added/updated
  • Integration tests added/updated
  • Documentation updated
  • Changelog updated

@blp blp self-assigned this Oct 30, 2026
@blp blp changed the title [connectors] Release the replay gates lock before calling a gate. [connectors] Unit test fix: Release the replay gates lock before calling a gate. Oct 30, 2026
@blp blp added bug Something isn't working connectors Issues related to the adapters/connectors crate CI/CD QA Testing and quality assurance labels Oct 30, 2026
@blp
blp added this pull request to the merge queue Oct 30, 2026
// endpoints can replay meanwhile.
let gate = REPLAY_GATES.lock().unwrap().remove(&self.name);
if let Some(gate) = gate {
gate();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pitfall (medium, flakiness): the gate still blocks a TOKIO worker thread, so the other endpoint's task can only run if some other worker is free. TOKIO_WORKER_THREADS=1 ./dbsp_connectors-… replay_gate_does_not_block fails after 10s. With 2 workers it passes. Any runner whose available_parallelism() is 1 (for example a container limited by a cgroup CPU quota) fails here and in replay_does_not_feed_http_input_during_commit. block_in_place hands this worker's core to another thread while the gate waits, so the test no longer depends on the worker count:

Suggested change
gate();
tokio::task::block_in_place(gate);

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Applied in 7cbe619: the gate now waits in block_in_place. replay_gate_does_not_block_other_endpoints failed with TOKIO_WORKER_THREADS=1 before the change, as you saw, and passes now; the transport::http::input tests pass 7 of 7 with 1 and with 2 workers. With the lock held again, the test still fails, so it still catches the original bug. replay_does_not_feed_http_input_during_commit passed in my one run with 1 worker even without block_in_place, so I could not reproduce that half; it passes with it too.

@blp
blp removed this pull request from the merge queue due to a manual request Oct 30, 2026
@mythical-fred-oss

Copy link
Copy Markdown
Contributor

The fix is correct and only affects tests. One flakiness gap remains (see the inline comment).

What I ran Result
cargo test --locked -p dbsp_connectors --lib transport::http::input 7/7 pass; 30 repeated runs, no failures
New test against the old if let code fails every time ("other endpoint could not replay") ✅
New test with TOKIO_WORKER_THREADS=1 fails after 10s (gate blocks the only worker)
Same, with tokio::task::block_in_place(gate) 7/7 pass
pre-commit run --files …/http/input.rs pass
  • pitfall (medium, flaky-test gate): the lock is now released, but the gate still blocks a tokio worker thread. So the new test and replay_does_not_feed_http_input_during_commit only pass when TOKIO has at least 2 workers. I'd take the one-line block_in_place change before merge.

`REPLAY_GATES` hooks hold an HTTP input endpoint's replay in tests.  The
endpoint's task called its gate inside `if let` on the locked map, so
the lock stayed held while the gate waited, and every other HTTP
endpoint's replay blocked on it.

The gate also blocked its tokio worker thread, so with one worker no
other endpoint could replay at all.  It now waits in `block_in_place`.

`replay_does_not_feed_http_input_during_commit` gates the second of two
endpoints until the first transaction commits.  When the second
endpoint's gate ran first, the first endpoint could not replay, so the
step never finished, and the test failed after its timeout.  That
happened once in about 70 runs of the controller suite;
`replay_gate_does_not_block_other_endpoints` shows it every time.

Signed-off-by: Ben Pfaff <blp@feldera.com>
@blp
blp force-pushed the replay-gate-lock branch from 88df2b6 to 7cbe619 Compare October 30, 2026 22:23
@blp
blp added this pull request to the merge queue Nov 2, 2026
Merged via the queue into main with commit 53b1842 Nov 2, 2026
1 check passed
@blp
blp deleted the replay-gate-lock branch November 2, 2026 20:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working CI/CD connectors Issues related to the adapters/connectors crate QA Testing and quality assurance

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants