Repository navigation
[connectors] Journal the input of an ad hoc INSERT removed before its step collects it. - #7672
Conversation
|
This is about the nth PR to fix related races. I'm going to follow up after this one with a refactoring that makes everything associated less bad. Here's Claude's take on the refactoring idea:
|
| // so this step processes it and must journal it. See | ||
| // `ControllerStatus::orphaned_results`. | ||
| if let Some(results) = | ||
| self.controller.status.take_orphaned_results(&endpoint_id) |
There was a problem hiding this comment.
pitfall (medium): This fixes the case where the reply comes before the removal. The reverse order still loses journal data. If an async transport is removed before its reply arrives, this branch finds no results and drops the endpoint from waiting. The late reply then goes through ControllerStatus::extended → orphan_input (stats.rs ~1398), which credits size only. Any rows that endpoint already pushed into the circuit get processed but never journaled. I see the stacked follow-up is meant to close this. Could you add a test for it there (reply after removal, then replay) so the follow-up has to keep that path covered?
There was a problem hiding this comment.
Yes, this is correct. If an endpoint is removed before its reply arrives, ControllerStatus::extended cannot find the endpoint. It then sends the input to orphan_input, which credits only its size, so the rows get to the circuit without a journal entry. This PR closes only the window where the reply comes first.
The stacked follow-up makes the circuit thread the only thing that removes input endpoints. That closes both orders. In the follow-up, I will add these tests:
- a replay test where the reply arrives after the endpoint is removed;
- results that no step claims, which
take_orphaned_inputcredits by size only; - an endpoint removed during
stop(), where the input is now journaled butwrite_stepskips the endpoint diff.
|
LGTM. The fix is correct. Lock order stays
Gates: tests ✅. Flakiness ✅: the waits are bounded and fail with clear messages. No docs needed (internal behaviour). No unsafe code, dependency changes or breaking changes. Missing (medium): nothing tests results that no step claims, which Nit: the commit subject is about 100 chars and GitHub truncates it. |
An ad hoc INSERT hands its rows to the circuit when a step asks its endpoint for input, then removes the endpoint as soon as its queue drains. If the removal came before the step collected the endpoint's reply, the step still processed the rows, but `remove_input` kept only their size, so the step's journal entry did not record them. A restart from the journal then lost rows whose INSERT had succeeded. `remove_input` now keeps the removed endpoint's full results, and the step that waits for the endpoint journals them like any other reply. Results that no step claims are still credited by size only. Found by an Antithesis run of the controller harness. Signed-off-by: Ben Pfaff <blp@feldera.com>
156c539 to
91bf1be
Compare
An ad hoc
INSERThands its rows to the circuit when a step asks its endpoint for input (reader.queue(), which the ad hoc transport handles synchronously on the circuit thread), and then removes its endpoint as soon as its queue drains. If the removal came before the step collected the endpoint's reply, the step still processed the rows, butControllerStatus::remove_inputkept only their size (fortotal_processed_records), so the step's journal entry did not record them. A restart from the journal then lost rows whoseINSERThad succeeded.remove_inputnow keeps the removed endpoint's fullStepResults, keyed by endpoint id, and stores them while it still holds theinputslock, so the circuit thread cannot see the endpoint gone without its results. The step that waits for the endpoint claims them and journals them like any other reply;write_stepalready handles journaled input from an endpoint removed after its step took its input. Results that no step claims are still credited by size only.Found by an Antithesis run of the controller harness.
A follow-up PR, stacked on this one, makes the circuit thread the only thing that removes input endpoints, which removes this window and several related special cases.
Describe Manual Test Plan
New test
adhoc_insert_removed_before_collection_replaysholds the step, through a new test hook that runs after the step asks its endpoints for input and before it collects their replies, until theINSERThas removed its endpoint, then restarts from the journal and checks that the row survives. Without the fix it fails with "the restart lost the INSERT's row". The controller tests pass (cargo test -p static-connectors --test connector_suites controller::, 3 runs, andcargo test -p dbsp_connectors --lib controller::), and the ad hoc tests passed 200 of 200 runs.Checklist