Repository navigation
[connectors] Fail, do not panic, when the pipeline stops during open. - #7785
Conversation
`insert_input_endpoint` registers a new input endpoint, drops the lock, calls the transport's `open`, and then sets the endpoint's reader. If `stop` (from `/suspend`, a restart, or shutdown) removed every input endpoint while `open` ran, setting the reader unwrapped a missing entry and panicked the thread that was adding the endpoint, such as an `/ingress` request or an ad hoc INSERT. It now disconnects the new reader and returns an error instead. Signed-off-by: Ben Pfaff <blp@feldera.com>
| Ok(()) | ||
| } | ||
| None => { | ||
| // `stop` removed the endpoint while `open` ran. |
There was a problem hiding this comment.
pitfall (medium): this fixes the case where stop runs during open, but not the one where it runs before the insert above. A QA test that calls initiate_stop() and then add_input_endpoint(...) gets Ok, and leaves 1 input with a live reader on a Terminated controller. stop only clears inputs once, so nothing disconnects that reader. Controller::stop happens to clear it again, but a path that only holds a ControllerInner clone (ingress, ad hoc INSERT) gets no second clear. One option: check Terminated under the insert's write lock and return this same error, which covers both orderings.
There was a problem hiding this comment.
Agreed. 1d8e36b adds the fix that you suggested: insert_input_endpoint now checks for Terminated while it holds the inputs write lock. stop sets Terminated before it takes that lock, so this check covers a stop before the insert, and the existing None branch covers a stop during open. Both cases now return one error, worded so that it fits both: "the pipeline stopped while the endpoint was being added".
| Check | Result |
|---|---|
New test add_input_endpoint_after_stop (initiate_stop(), then add_input_endpoint(), expect Err and 0 inputs) |
passes |
Same test with the Terminated check disabled |
fails at the is_err assert |
controller::test suite |
94/94 pass |
(Claude, on behalf of Ben)
|
LGTM. The fix is correct and the test is deterministic:
|
…ops. `stop` clears the input endpoints only once. An endpoint added after that kept a live reader on a terminated controller, because paths that hold only a `ControllerInner` (ingress, ad hoc INSERT) never clear the inputs again. `insert_input_endpoint` now checks for `Terminated` under the inputs lock, which covers a stop before the insert as well as one during `open`. Signed-off-by: Ben Pfaff <blp@feldera.com>
insert_input_endpointregisters a new input endpoint, drops the lock, calls the transport'sopen, and then sets the endpoint's reader. Ifstop(from/suspend, a restart, or shutdown) removed every input endpoint whileopenran, setting the reader unwrapped a missing entry and panicked the thread that was adding the endpoint, such as an/ingressrequest or an ad hoc INSERT.It now disconnects the new reader and returns an error instead.
stop_during_input_endpoint_open(new)unwrapFound while building the Antithesis property
steps-progress-despite-endpoint-hot-add.