Repository navigation
[connectors] Unit test fix: Release the replay gates lock before calling a gate. - #7753
Conversation
| // endpoints can replay meanwhile. | ||
| let gate = REPLAY_GATES.lock().unwrap().remove(&self.name); | ||
| if let Some(gate) = gate { | ||
| gate(); |
There was a problem hiding this comment.
pitfall (medium, flakiness): the gate still blocks a TOKIO worker thread, so the other endpoint's task can only run if some other worker is free. TOKIO_WORKER_THREADS=1 ./dbsp_connectors-… replay_gate_does_not_block fails after 10s. With 2 workers it passes. Any runner whose available_parallelism() is 1 (for example a container limited by a cgroup CPU quota) fails here and in replay_does_not_feed_http_input_during_commit. block_in_place hands this worker's core to another thread while the gate waits, so the test no longer depends on the worker count:
| gate(); | |
| tokio::task::block_in_place(gate); |
There was a problem hiding this comment.
Applied in 7cbe619: the gate now waits in block_in_place. replay_gate_does_not_block_other_endpoints failed with TOKIO_WORKER_THREADS=1 before the change, as you saw, and passes now; the transport::http::input tests pass 7 of 7 with 1 and with 2 workers. With the lock held again, the test still fails, so it still catches the original bug. replay_does_not_feed_http_input_during_commit passed in my one run with 1 worker even without block_in_place, so I could not reproduce that half; it passes with it too.
|
The fix is correct and only affects tests. One flakiness gap remains (see the inline comment).
|
`REPLAY_GATES` hooks hold an HTTP input endpoint's replay in tests. The endpoint's task called its gate inside `if let` on the locked map, so the lock stayed held while the gate waited, and every other HTTP endpoint's replay blocked on it. The gate also blocked its tokio worker thread, so with one worker no other endpoint could replay at all. It now waits in `block_in_place`. `replay_does_not_feed_http_input_during_commit` gates the second of two endpoints until the first transaction commits. When the second endpoint's gate ran first, the first endpoint could not replay, so the step never finished, and the test failed after its timeout. That happened once in about 70 runs of the controller suite; `replay_gate_does_not_block_other_endpoints` shows it every time. Signed-off-by: Ben Pfaff <blp@feldera.com>
REPLAY_GATEShooks hold an HTTP input endpoint's replay in tests. The endpoint's task called its gate insideif leton the locked map, so the lock stayed held while the gate waited (in edition 2024, too, a temporary in anif letscrutinee lives through thethenblock), and every other HTTP endpoint's replay blocked on it.replay_does_not_feed_http_input_during_commitgates the second of two endpoints until the first transaction commits. When the second endpoint's gate ran first, the first endpoint could not replay, so the step never finished, and the test failed after its timeout ("replay never finished"). That happened once in about 70 runs of the controller suite.The task now takes the gate out of the map, releasing the lock, before it calls the gate. The gate also blocked its tokio worker thread, so with one worker (
TOKIO_WORKER_THREADS=1, or a runner whoseavailable_parallelism()is 1) no other endpoint could replay at all; it now waits inblock_in_place. Test-only code; no production path changes.Describe Manual Test Plan
New unit test
replay_gate_does_not_block_other_endpoints: one endpoint's gate waits for a second endpoint to finish its replay. It fails every time with the old code ("the other endpoint could not replay while the gate waited") and passes with the fix, and it also fails withTOKIO_WORKER_THREADS=1withoutblock_in_place.cargo test -p dbsp_connectors --lib transport::http::inputpasses 7 of 7 with 1 and with 2 tokio workers, and clippy is clean.Checklist