← 输入框|github.com|刷新
Skip to content

[connectors] Remove disconnected input endpoints only between steps. - #7680

Merged
blp merged 5 commits into
mainfrom
deferred-input-removal
Nov 10, 2026
Merged

blp merged 5 commits into
mainfrom
deferred-input-removal

Conversation

@blp

@blp blp commented Oct 28, 2026 •

Copy link
Copy Markdown
Member

Any thread could remove an input endpoint at any time, for example an ad hoc INSERT as soon as its queue drained. A step's journal entry takes the list of endpoints from the input status when it is written, after the step, and the input from what the step collected, so a removal in between made the two disagree. Each such window has needed its own repair:

Window Repair before this PR
Removed after it handed input to the circuit, before the step counted it orphaned_input credits its size
Removed before the step collected its reply orphaned_results (#7672) journals it
Removed after collection, before the journal write write_step keeps a logged endpoint in the journal's endpoint list
Added and removed within one step pending_new_inputs re-adds it

disconnect_input now disconnects the endpoint's reader and marks the endpoint for removal. The circuit thread removes it between steps, once a step has collected any input that it already reported (CircuitThread::remove_requested_inputs). Only the circuit thread changes which input endpoints exist, and only between steps, so the endpoints that a step's journal entry records are the ones that supplied its input. This drops the #7672 stash and the logged retention, and it closes both orders that the #7672 review raised: the reply before the removal, and the reply after it. A step keeps waiting for an endpoint that it asked for input even after the endpoint is disconnected; InputReader::is_closed documents that a reader acts on its commands in order, so it answers the step's Queue or closes. orphaned_input stays as a safety net, so that total_processed_records cannot fall permanently behind total_input_records and stall wait_for_idle or pipeline_complete. Outside stop, only a reader that breaks the contract can report after its removal, so the controller now logs a warning when that happens. pending_new_inputs stays for stop, which still removes every endpoint at once.

Other details:

  • The HTTP input reader acts on commands on a tokio task. It now reports is_closed when that task is gone, as the InputReader docs suggest, because a step waits for a disconnected endpoint's answer. The is_closed docs now say so.
  • A marked endpoint is not polled for more input. A step still collects what it already reported, and disconnect_input and extended request that step.
  • The backpressure thread does not restart a marked endpoint.
  • The circuit thread removes marked endpoints before it starts a checkpoint, so that a checkpoint does not record a disconnected endpoint.
  • Removals made by the circuit thread itself (replay, failed endpoints, restored ad hoc endpoints) still happen at once, through remove_input_endpoint.
  • A disconnected endpoint stays visible in the status and metrics until the next step boundary.

Describe Manual Test Plan

New tests, each checked to fail under the change named:

Test What it does Fails when
input_disconnected_before_journaling_replays disconnects an HTTP endpoint, which an earlier entry journaled, after a step collects its input but before the step is journaled; restarts disconnect_input removes at once, as before ("unknown input endpoint name 'http'")
input_reported_after_disconnect_replays a reader that acts on commands on its own thread (QueueGatedEndpoint) answers the step's Queue only after disconnect_input; restarts removal is immediate, or the step stops waiting for a disconnected endpoint ("the restart lost the rows reported after the disconnect")
input_collected_while_stopping_replays stop removes a new endpoint after its step collected input, before the journal write; restarts the journal ignores pending_new_inputs ("unknown input endpoint name 'late'")
input_disconnected_while_paused_is_removed_without_step disconnects an endpoint while the pipeline is paused the circuit thread does not remove marked endpoints at the top of its loop
input_disconnected_before_checkpoint_is_not_restored disconnects an endpoint, checkpoints, restarts from the checkpoint neither the loop nor the checkpoint removes marked endpoints ("the restart brought back the disconnected endpoint")
disconnected_input_is_not_restarted starts a disconnected endpoint; a second endpoint witnesses the backpressure pass the backpressure thread does not skip marked endpoints
endpoint_is_closed_after_its_task_dies (dbsp_connectors lib) kills the HTTP reader's task with bad replay data HttpInputEndpoint::is_closed returns false

removed_input_endpoint_input_is_still_counted now waits until the endpoint is gone before its late report, so it always covers the size-only credit in orphaned_input; without that credit it fails. The existing W1 and #7672 tests now hold the step until the INSERT has disconnected its endpoint, not until the endpoint is gone; the #7672 test fails if disconnect_input removes at once.

The controller suite (cargo test -p static-connectors --test connector_suites controller::) passes 98 of 98 in 9 of 10 runs. The other run failed in replay_does_not_feed_http_input_during_commit, from a deadlock in the REPLAY_GATES test hook that main has too (it calls the gate with the gates lock held); I will fix that separately. cargo test -p dbsp_connectors --lib passes except the same 26 Kafka, Postgres and DynamoDB tests that fail without this change here (they need those services).

Checklist

  • Unit tests added/updated
  • Integration tests added/updated
  • Documentation updated
  • Changelog updated

@blp
blp force-pushed the deferred-input-removal branch 2 times, most recently from ed670cf to 674abfb Compare October 29, 2026 20:32
Base automatically changed from adhoc-orphan-journal to main October 29, 2026 21:10
@blp
blp force-pushed the deferred-input-removal branch from 674abfb to 93607e9 Compare October 29, 2026 21:22
@blp blp self-assigned this Oct 29, 2026
@blp blp added bug Something isn't working connectors Issues related to the adapters/connectors crate QA Testing and quality assurance rust Pull requests that update Rust code labels Oct 29, 2026
@blp
blp force-pushed the deferred-input-removal branch 3 times, most recently from 143838b to a3a64a9 Compare October 29, 2026 23:00
}
return false;
} else {
// Keep waiting even if the endpoint was disconnected

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pitfall (medium): Before this PR, removing an endpoint always ended the step's wait for it (statuses.get → None). Now the step waits for a disconnected endpoint until it answers or is_closed() returns true. HTTP and ad hoc readers hard-code is_closed() == false, so if their worker task has exited (for example after a panic in background_task), the circuit thread now waits forever and disconnect_input can no longer break the wait. Consider a bound or a warning (as LongOperationWarning does elsewhere), or at least spell out this new liveness dependency on InputReader::is_closed in its docs.

Comment thread crates/connectors/src/controller.rs Outdated
// barrier.
self.set_checkpoint_coordination(Some(CheckpointCoordination::InProgress));

// Don't checkpoint an endpoint that was disconnected.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bug (low-medium): remove_requested_inputs skips an endpoint whose progress is still Some, so the checkpoint can still record a disconnected endpoint, and the comment overstates the guarantee. On resume, a non-ad hoc endpoint comes back. Either reword the comment ("…unless it still holds uncollected input") or wait for that input to be collected before starting the checkpoint. No test covers disconnecting an endpoint and then checkpointing.

@mythical-fred-oss

Copy link
Copy Markdown
Contributor

Review: remove disconnected input endpoints only between steps

The design is sound: only the circuit thread now changes which input endpoints exist. I checked the flag/progress handshake between disconnect_input and extended. The mutex orders it, so no request for a step gets lost. I also checked that the in-tree readers (file, url, s3, kafka-ft, nats, http, adhoc, delta) act on their commands in order or close.

What I ran

Command Result
cargo test --locked -p static-connectors --test connector_suites controller:: ✅ 95/95
cargo test --locked -p dbsp_connectors --lib controller::test ✅ 36/36
rustfmt --check on changed files ✅

Missing test coverage (medium)

  • Disconnecting while the pipeline is paused, for an endpoint with no pending progress: it should be removed on the next loop iteration, with no step.
  • Calling disconnect_input twice: request_removal returns early, and the transaction should end only once.
  • Disconnecting and then checkpointing (see inline comment).
  • The backpressure thread must not restart a marked endpoint, including after start_input_endpoint is called on it.

Pitfall (low, flaky-adjacent)

  • input_reported_after_disconnect_replays uses a 500 ms negative wait. It cannot fail spuriously, but on a slow CI runner it can pass without exercising the regression it targets.

Commit hygiene (low)

  • The subject ends with a period.
  • One body line is longer than 72 characters.

@blp
blp force-pushed the deferred-input-removal branch from a3a64a9 to 64f9759 Compare October 30, 2026 00:38
@blp

blp commented Oct 30, 2026

Copy link
Copy Markdown
Member Author

Thanks. I pushed changes for each point:

Point Change
Wait on a dead HTTP worker (inline) Real: the HTTP reader runs its commands on a tokio task but always said is_closed() == false. It now returns sender.is_closed(), as the InputReader::is_closed docs suggest, so a task that panicked closes the reader and ends the wait. New unit test endpoint_is_closed_after_its_task_dies (fails with the old false). The ad hoc reader acts on Queue synchronously inside request, so it cannot leave a step waiting. The is_closed docs now state the liveness dependency. Before this PR the dependency was already there for every endpoint that was not removed; removal was the only way out.
Checkpoint comment (inline) Reworded: an endpoint that still holds uncollected input, or that is disconnected after the boundary, is in the checkpoint, and the journal entry of the first step after its removal records the removal. That matches main, where a removal right after the checkpoint's snapshot did the same.
Disconnect while paused input_disconnected_while_paused_is_removed_without_step: removed with no step. Fails without the removal at the top of the loop.
Disconnect, then checkpoint input_disconnected_before_checkpoint_is_not_restored: restart from the checkpoint does not bring the endpoint back, and keeps its rows. Fails when neither the loop nor the checkpoint removes marked endpoints.
Backpressure disconnected_input_is_not_restarted: after disconnect_input and start_input_endpoint, the backpressure thread sends no Extend. A second endpoint with a higher id is the witness that the backpressure pass ran, so there is no negative wait. Fails without the skip in the backpressure thread.
Double disconnect_input No test: the second call is harmless even without the request_removal guard, because commit_transaction_from_connector returns NoTransactionInProgress for a connector that already committed (its phase is no longer Started, or on multihost its entry is gone), so end_transaction_for_stopped_connector does nothing. A test would pass either way.
500 ms negative wait Agreed that it can pass without exercising the regression on a slow runner; it cannot fail spuriously. The correct controller has no event to wait for there (the step is blocked on the gated reply), so I kept it. It failed 3 of 3 runs here with the early exit restored.
Subject ends with a period Kept; the repository has no rule on it, and #7672's subject has one too.
Body line longer than 72 Fixed in an earlier push; every line is now 72 or less.

@blp
blp requested a review from ryzhyk October 30, 2026 22:17
@ryzhyk

ryzhyk commented Nov 6, 2026

Copy link
Copy Markdown
Contributor

This PR introduces a new contract:

      /// The controller waits for an endpoint to act on a command such as
      /// [InputReaderCommand::Queue] until it acts on it or closes, even after
      /// it asks the endpoint to [InputReaderCommand::Disconnect].  An endpoint
      /// that stops acting on commands without closing makes the controller wait
      /// forever.

I asked claude whether any of the connectors violate the contract and it claims to have found violations:

Contract audit (17 production readers)

┌─────────────────────────────────────────────────────────┬────────────────────────────────────────────────────────────────────────────┬────────────────────────────────────┐
│                         Reader                          │                            How it answers Queue                            │              Verdict               │
├─────────────────────────────────────────────────────────┼────────────────────────────────────────────────────────────────────────────┼────────────────────────────────────┤
│ ad hoc, Delta, Iceberg, Postgres, Postgres CDC, empty,  │ synchronously, inside request                                              │ conform                            │
│ Nexmark Person/Auction                                  │                                                                            │                                    │
├─────────────────────────────────────────────────────────┼────────────────────────────────────────────────────────────────────────────┼────────────────────────────────────┤
│ file, URL, S3, NATS, S2, Pub/Sub, datagen               │ a worker takes commands in order from one channel; every exit drops the    │ conform                            │
│                                                         │ receiver, so is_closed turns true                                          │                                    │
├─────────────────────────────────────────────────────────┼────────────────────────────────────────────────────────────────────────────┼────────────────────────────────────┤
│ Kafka FT                                                │ same as above                                                              │ Queue conforms; replay can hang    │
│                                                         │                                                                            │ (read, not run)                    │
├─────────────────────────────────────────────────────────┼────────────────────────────────────────────────────────────────────────────┼────────────────────────────────────┤
│ HTTP                                                    │ a tokio task, in order                                                     │ Queue conforms; replay violates    │
│                                                         │                                                                            │ (observed)                         │
├─────────────────────────────────────────────────────────┼────────────────────────────────────────────────────────────────────────────┼────────────────────────────────────┤
│ clock                                                   │ a tokio task, in order                                                     │ Queue conforms; replay violates    │
│                                                         │                                                                            │ (observed)                         │
├─────────────────────────────────────────────────────────┼────────────────────────────────────────────────────────────────────────────┼────────────────────────────────────┤
│ Nexmark Bid                                             │ a worker thread, in order                                                  │ violates (observed)                │
└─────────────────────────────────────────────────────────┴────────────────────────────────────────────────────────────────────────────┴────────────────────────────────────┘

Each violation stalls the pipeline until stop. A fatal input error never stops a pipeline: server::is_fatal_controller_error covers only DBSP errors and panics.

1. HTTP. In HttpInputEndpointInner::background_task (crates/connectors/src/transport/http/input.rs:179), a replay of journaled connector_metadata that this version refuses reports a fatal error. It then returns from the closure only, so the task lives on: no replayed, and is_closed() stays false. The PR's is_closed fix covers only a task that panics. This path came in with 263ad25bf; the existing test replay_of_refused_metadata_fails_the_endpoint checks only the error message.
   - Probe output: n_replayed=0 is_closed=false answers_later_queue=true.
2. Clock. In ClockReader::worker_task (crates/connectors/src/transport/clock.rs:319), a non-integer replay timestamp leads to a fatal error and continue.
   - Probe output: n_replayed=0 is_closed=false worker_alive=true.
3. Nexmark Bid. nexmark::InputGenerator::is_closed always returns false (crates/static-connectors/src/nexmark.rs:141), yet nexmark::worker_thread exits on ? (line 338).
   - Probe output: after undecodable replay metadata, a later Queue gets no answer and is_closed stays false.
   - From the code (not run): a panic in a generator thread also strands the worker in Barrier::wait.
4. Kafka FT (read, not run). The replay loop in KafkaFtInputReaderInner::poller_thread marks a partition done only when it reads exactly offset == max (crates/connectors/src/transport/kafka/ft/input.rs:637). If compaction or retention removed that offset, the loop never ends. NATS (consume_nats_messages_until) and S2 (replay_checkpoint) treat the same gap as a fatal error.

Fixes:
- HTTP and clock: exit the task or loop after the fatal error, so the controller fails the replay with "input reader closed before completing replay".
- Nexmark: have the Bid reader report command_sender.is_closed().
- Kafka: fail the replay when next_offset > max without having seen max.

@ryzhyk

ryzhyk commented Nov 6, 2026

Copy link
Copy Markdown
Contributor
PR-introduced race: commands after Disconnect (from the code, not demonstrated)

ControllerInner::disconnect_input now holds only a read guard (crates/connectors/src/controller.rs:8317). Two other places also hold only a read guard between checking the removal flag and sending a command:
- CircuitThread::input_step, between the flag check (line 4659) and reader.queue (line 4679).
- BackpressureThread::backpressure_thread, between its check (line 6637) and extend/pause.

Read guards don't exclude each other, so Queue, Extend or Pause can arrive after Disconnect. On main, ControllerStatus::remove_input took the write lock before disconnect was sent, which ruled this out.

This breaks the documented promise on InputReaderCommand::Disconnect and the PR's own comment "it acts on our Queue before the Disconnect". It is harmless for production readers today, but Nexmark Bid would hang, and a late Extend would set HTTP's state back to Running.

Fix: set the flag under inputs.write(), and send Disconnect after releasing the lock.

@ryzhyk

ryzhyk commented Nov 6, 2026

Copy link
Copy Markdown
Contributor
- Minor, from the PR: StepSelection::polls still selects marked endpoints. The step trigger therefore counts their buffered records, which input_step will no longer collect. The StepSelection docs say exactly this mismatch makes the circuit take empty steps. Here it costs about one empty step per disconnect, because the endpoint is removed at the next loop top. Fix: make polls return false for marked endpoints.

@ryzhyk ryzhyk left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Controller-level changes look sound, but it looks like not all connectors respect new protocol requirements.

blp added 5 commits November 9, 2026 12:44
The controller waits for an input endpoint to act on a command until
it acts on it or closes.  The HTTP endpoint broke this in two ways:

- A replay of journaled connector metadata that this version refuses
  reported a fatal error and then returned only from the closure for
  that command.  The task lived on, so the replay never finished.

- `is_closed` always returned `false`, so even a task that ended, for
  example after a panic, left the controller waiting forever.

The task now exits after the fatal error, and `is_closed` reports
whether the task's channel is closed.

Signed-off-by: Ben Pfaff <blp@feldera.com>
A replay of a timestamp that is not an integer reported a fatal error
and then went on to the next command, so the replay never finished
and the endpoint never closed.  The controller waits for an input
endpoint to act on a command until it acts on it or closes, so it
waited forever.  The worker now exits after the fatal error.

Signed-off-by: Ben Pfaff <blp@feldera.com>
The Bid endpoint's worker thread exits on a fatal error, for example
when replay metadata does not decode, but `is_closed` always returned
`false`.  The controller waits for an input endpoint to act on a
command until it acts on it or closes, so it waited forever.  The Bid
endpoint now reports whether the worker's channel is closed.  The
other tables act on each command inside `request`, so they stay open.

Signed-off-by: Ben Pfaff <blp@feldera.com>
A replay step waited until it read the message at the last journaled
offset of each partition.  If the topic no longer held that offset, the
step waited forever, and the controller with it:

- Retention or compaction deleted the offset.  The reader then read
  past it without seeing it.

- The topic was deleted and created again, so the partition ends
  before the offset.

The replay now fails in both cases.  The first shows as the reader's
next offset going past the last journaled offset without that message.
For this check to be sound, the reader now advances its next offset
only after it stores the message.  The second shows as a step that
makes no progress for a second while the broker's high watermark is at
or below the last journaled offset.

Signed-off-by: Ben Pfaff <blp@feldera.com>
Any thread could remove an input endpoint at any time, for example an ad
hoc INSERT as soon as its queue drained.  A step's journal entry derives
which endpoints exist from the input status when it is written, and the
input from what the step collected, so a removal in between made the two
disagree.  Each such window needed its own repair: crediting the input
of an endpoint removed before collection, journaling it, and keeping an
endpoint that was removed after collection in the journal's endpoint
list.

`disconnect_input` now disconnects the endpoint's reader and marks it
for removal, and the circuit thread removes it between steps, once a
step has collected the input it already reported.  A step keeps waiting
for an endpoint that it asked for input even after the endpoint is
disconnected, because the reader acts on the `Queue` before the
`Disconnect`.  The endpoints that a step's journal entry records are
then the ones that supplied its input, so the repairs for removal during
a step go away.  `stop` still removes every endpoint at once, which
`pending_new_inputs` continues to handle.

`disconnect_input` marks the endpoint under the write lock on the input
status.  The circuit thread and the backpressure thread check the mark
and send commands under the read lock, so a reader always gets their
`Queue`, `Extend`, or `Pause` before the `Disconnect`.  The step trigger
does not count input buffered by a marked endpoint, because no step
asks it for that input.

The circuit thread also removes marked endpoints before it starts a
checkpoint, so that a checkpoint does not record a disconnected
endpoint.

`orphaned_input` still credits input reported to an endpoint that is
already gone.  Only `stop`, or a reader that breaks the `InputReader`
contract, can do that, and the controller now warns about the latter.

Signed-off-by: Ben Pfaff <blp@feldera.com>
@blp
blp force-pushed the deferred-input-removal branch from 64f9759 to 03693b7 Compare November 10, 2026 00:08
@blp

blp commented Nov 10, 2026

Copy link
Copy Markdown
Member Author

@ryzhyk Thanks. I rebased and changed the commits. The connector bugs were already on main, so each fix is a commit of its own before the PR's commit.

Point Change
HTTP replay The task exits after the fatal error, so the endpoint closes. replay_of_refused_metadata_fails_the_endpoint now checks is_closed().
clock replay The worker exits after the fatal error. New unit test.
Nexmark Bid is_closed() returns command_sender.is_closed(). New unit test. A panic in a generator thread can still leave the worker stuck in Barrier::wait; that is not changed here.
Kafka FT replay gap The replay fails when the reader reads past the last journaled offset without that message, as after retention or compaction. It also fails when a step makes no progress for a second and the broker's high watermark is at or below that offset, as after the topic is deleted and created again. A check of next_offset > max alone misses the second case. Partition EOF could not do it either, because it is off by default. For the first check to be sound, the reader now advances next_offset only after it stores the message. There are two new tests, one for each case, and both fail without the fix.
Commands after Disconnect disconnect_input sets the flag under inputs.write() and sends Disconnect after it releases the lock. New test input_disconnected_while_queued_gets_queue_first holds a step inside request(Queue). It fails 3 of 3 runs with a read lock.
polls Returns false for endpoints marked for removal. disconnected_input_is_not_restarted now checks this, and it fails without the change.

(Claude)

@blp
blp added this pull request to the merge queue Nov 10, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Nov 10, 2026
@blp
blp added this pull request to the merge queue Nov 10, 2026
Merged via the queue into main with commit 89fc01f Nov 10, 2026
1 check passed
@blp
blp deleted the deferred-input-removal branch November 10, 2026 19:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working connectors Issues related to the adapters/connectors crate QA Testing and quality assurance rust Pull requests that update Rust code

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants