← 输入框|github.com|刷新
Skip to content

[connectors] Fail, do not panic, when the pipeline stops during open. - #7785

Merged
blp merged 2 commits into
mainfrom
stop-during-open
Nov 3, 2026
Merged

blp merged 2 commits into
mainfrom
stop-during-open

Conversation

@blp

@blp blp commented Nov 2, 2026

Copy link
Copy Markdown
Member

insert_input_endpoint registers a new input endpoint, drops the lock, calls the transport's open, and then sets the endpoint's reader. If stop (from /suspend, a restart, or shutdown) removed every input endpoint while open ran, setting the reader unwrapped a missing entry and panicked the thread that was adding the endpoint, such as an /ingress request or an ad hoc INSERT.

It now disconnects the new reader and returns an error instead.

Test Before the fix After the fix
stop_during_input_endpoint_open (new) panics at the unwrap passes
controller suite 93/93, 3 runs

Found while building the Antithesis property steps-progress-despite-endpoint-hot-add.

`insert_input_endpoint` registers a new input endpoint, drops the lock,
calls the transport's `open`, and then sets the endpoint's reader.  If
`stop` (from `/suspend`, a restart, or shutdown) removed every input
endpoint while `open` ran, setting the reader unwrapped a missing entry
and panicked the thread that was adding the endpoint, such as an
`/ingress` request or an ad hoc INSERT.

It now disconnects the new reader and returns an error instead.

Signed-off-by: Ben Pfaff <blp@feldera.com>
@blp blp self-assigned this Nov 2, 2026
@blp blp added bug Something isn't working connectors Issues related to the adapters/connectors crate QA Testing and quality assurance rust Pull requests that update Rust code labels Nov 2, 2026
@blp
blp enabled auto-merge November 2, 2026 18:41
Ok(())
}
None => {
// `stop` removed the endpoint while `open` ran.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pitfall (medium): this fixes the case where stop runs during open, but not the one where it runs before the insert above. A QA test that calls initiate_stop() and then add_input_endpoint(...) gets Ok, and leaves 1 input with a live reader on a Terminated controller. stop only clears inputs once, so nothing disconnects that reader. Controller::stop happens to clear it again, but a path that only holds a ControllerInner clone (ingress, ad hoc INSERT) gets no second clear. One option: check Terminated under the insert's write lock and return this same error, which covers both orderings.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed. 1d8e36b adds the fix that you suggested: insert_input_endpoint now checks for Terminated while it holds the inputs write lock. stop sets Terminated before it takes that lock, so this check covers a stop before the insert, and the existing None branch covers a stop during open. Both cases now return one error, worded so that it fits both: "the pipeline stopped while the endpoint was being added".

Check Result
New test add_input_endpoint_after_stop (initiate_stop(), then add_input_endpoint(), expect Err and 0 inputs) passes
Same test with the Terminated check disabled fails at the is_err assert
controller::test suite 94/94 pass

(Claude, on behalf of Ben)

@mythical-fred-oss

Copy link
Copy Markdown
Contributor

LGTM. The fix is correct and the test is deterministic: initiate_stop clears inputs synchronously before open is released, and the test uses channels with no sleeps.

What I ran Result
cargo test -p static-connectors --test connector_suites -- stop_during_input_endpoint_open endpoint_without_reader, 5 runs 5/5 pass, ~0.14s
Same test with the stats.rs change reverted fails, panics at stats.rs:1803 (the unwrap), so the test catches the bug
pre-commit run --files on both files all pass
QA probe: initiate_stop(), then add_input_endpoint() returns Ok and registers 1 input on a terminated controller (see inline comment)
  • missing (medium): no test covers stop-before-insert. The inline comment has the details; a fix that checks for Terminated under the lock would also need this test.
  • No feldera-pr-checks gate fails: tests are added, no unsafe, no deps, no docs or user-visible API changes.

@blp
blp disabled auto-merge November 2, 2026 19:24
@blp
blp added this pull request to the merge queue Nov 2, 2026
…ops.

`stop` clears the input endpoints only once.  An endpoint added after
that kept a live reader on a terminated controller, because paths that
hold only a `ControllerInner` (ingress, ad hoc INSERT) never clear the
inputs again.  `insert_input_endpoint` now checks for `Terminated` under
the inputs lock, which covers a stop before the insert as well as one
during `open`.

Signed-off-by: Ben Pfaff <blp@feldera.com>
@blp
blp removed this pull request from the merge queue due to a manual request Nov 2, 2026
@blp
blp added this pull request to the merge queue Nov 2, 2026
Merged via the queue into main with commit 7e3bb7f Nov 3, 2026
1 check passed
@blp
blp deleted the stop-during-open branch November 3, 2026 00:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working connectors Issues related to the adapters/connectors crate QA Testing and quality assurance rust Pull requests that update Rust code

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants