Repository navigation
[connectors] Remove disconnected input endpoints only between steps. - #7680
Conversation
ed670cf to
674abfb
Compare
674abfb to
93607e9
Compare
143838b to
a3a64a9
Compare
| } | ||
| return false; | ||
| } else { | ||
| // Keep waiting even if the endpoint was disconnected |
There was a problem hiding this comment.
pitfall (medium): Before this PR, removing an endpoint always ended the step's wait for it (statuses.get → None). Now the step waits for a disconnected endpoint until it answers or is_closed() returns true. HTTP and ad hoc readers hard-code is_closed() == false, so if their worker task has exited (for example after a panic in background_task), the circuit thread now waits forever and disconnect_input can no longer break the wait. Consider a bound or a warning (as LongOperationWarning does elsewhere), or at least spell out this new liveness dependency on InputReader::is_closed in its docs.
| // barrier. | ||
| self.set_checkpoint_coordination(Some(CheckpointCoordination::InProgress)); | ||
|
|
||
| // Don't checkpoint an endpoint that was disconnected. |
There was a problem hiding this comment.
bug (low-medium): remove_requested_inputs skips an endpoint whose progress is still Some, so the checkpoint can still record a disconnected endpoint, and the comment overstates the guarantee. On resume, a non-ad hoc endpoint comes back. Either reword the comment ("…unless it still holds uncollected input") or wait for that input to be collected before starting the checkpoint. No test covers disconnecting an endpoint and then checkpointing.
|
Review: remove disconnected input endpoints only between steps The design is sound: only the circuit thread now changes which input endpoints exist. I checked the flag/ What I ran
Missing test coverage (medium)
Pitfall (low, flaky-adjacent)
Commit hygiene (low)
|
a3a64a9 to
64f9759
Compare
|
Thanks. I pushed changes for each point:
|
|
This PR introduces a new contract: I asked claude whether any of the connectors violate the contract and it claims to have found violations: |
|
|
ryzhyk
left a comment
There was a problem hiding this comment.
Controller-level changes look sound, but it looks like not all connectors respect new protocol requirements.
The controller waits for an input endpoint to act on a command until it acts on it or closes. The HTTP endpoint broke this in two ways: - A replay of journaled connector metadata that this version refuses reported a fatal error and then returned only from the closure for that command. The task lived on, so the replay never finished. - `is_closed` always returned `false`, so even a task that ended, for example after a panic, left the controller waiting forever. The task now exits after the fatal error, and `is_closed` reports whether the task's channel is closed. Signed-off-by: Ben Pfaff <blp@feldera.com>
A replay of a timestamp that is not an integer reported a fatal error and then went on to the next command, so the replay never finished and the endpoint never closed. The controller waits for an input endpoint to act on a command until it acts on it or closes, so it waited forever. The worker now exits after the fatal error. Signed-off-by: Ben Pfaff <blp@feldera.com>
The Bid endpoint's worker thread exits on a fatal error, for example when replay metadata does not decode, but `is_closed` always returned `false`. The controller waits for an input endpoint to act on a command until it acts on it or closes, so it waited forever. The Bid endpoint now reports whether the worker's channel is closed. The other tables act on each command inside `request`, so they stay open. Signed-off-by: Ben Pfaff <blp@feldera.com>
A replay step waited until it read the message at the last journaled offset of each partition. If the topic no longer held that offset, the step waited forever, and the controller with it: - Retention or compaction deleted the offset. The reader then read past it without seeing it. - The topic was deleted and created again, so the partition ends before the offset. The replay now fails in both cases. The first shows as the reader's next offset going past the last journaled offset without that message. For this check to be sound, the reader now advances its next offset only after it stores the message. The second shows as a step that makes no progress for a second while the broker's high watermark is at or below the last journaled offset. Signed-off-by: Ben Pfaff <blp@feldera.com>
Any thread could remove an input endpoint at any time, for example an ad hoc INSERT as soon as its queue drained. A step's journal entry derives which endpoints exist from the input status when it is written, and the input from what the step collected, so a removal in between made the two disagree. Each such window needed its own repair: crediting the input of an endpoint removed before collection, journaling it, and keeping an endpoint that was removed after collection in the journal's endpoint list. `disconnect_input` now disconnects the endpoint's reader and marks it for removal, and the circuit thread removes it between steps, once a step has collected the input it already reported. A step keeps waiting for an endpoint that it asked for input even after the endpoint is disconnected, because the reader acts on the `Queue` before the `Disconnect`. The endpoints that a step's journal entry records are then the ones that supplied its input, so the repairs for removal during a step go away. `stop` still removes every endpoint at once, which `pending_new_inputs` continues to handle. `disconnect_input` marks the endpoint under the write lock on the input status. The circuit thread and the backpressure thread check the mark and send commands under the read lock, so a reader always gets their `Queue`, `Extend`, or `Pause` before the `Disconnect`. The step trigger does not count input buffered by a marked endpoint, because no step asks it for that input. The circuit thread also removes marked endpoints before it starts a checkpoint, so that a checkpoint does not record a disconnected endpoint. `orphaned_input` still credits input reported to an endpoint that is already gone. Only `stop`, or a reader that breaks the `InputReader` contract, can do that, and the controller now warns about the latter. Signed-off-by: Ben Pfaff <blp@feldera.com>
64f9759 to
03693b7
Compare
|
@ryzhyk Thanks. I rebased and changed the commits. The connector bugs were already on main, so each fix is a commit of its own before the PR's commit.
(Claude) |
Any thread could remove an input endpoint at any time, for example an ad hoc
INSERTas soon as its queue drained. A step's journal entry takes the list of endpoints from the input status when it is written, after the step, and the input from what the step collected, so a removal in between made the two disagree. Each such window has needed its own repair:orphaned_inputcredits its sizeorphaned_results(#7672) journals itwrite_stepkeeps aloggedendpoint in the journal's endpoint listpending_new_inputsre-adds itdisconnect_inputnow disconnects the endpoint's reader and marks the endpoint for removal. The circuit thread removes it between steps, once a step has collected any input that it already reported (CircuitThread::remove_requested_inputs). Only the circuit thread changes which input endpoints exist, and only between steps, so the endpoints that a step's journal entry records are the ones that supplied its input. This drops the #7672 stash and theloggedretention, and it closes both orders that the #7672 review raised: the reply before the removal, and the reply after it. A step keeps waiting for an endpoint that it asked for input even after the endpoint is disconnected;InputReader::is_closeddocuments that a reader acts on its commands in order, so it answers the step'sQueueor closes.orphaned_inputstays as a safety net, so thattotal_processed_recordscannot fall permanently behindtotal_input_recordsand stallwait_for_idleorpipeline_complete. Outsidestop, only a reader that breaks the contract can report after its removal, so the controller now logs a warning when that happens.pending_new_inputsstays forstop, which still removes every endpoint at once.Other details:
is_closedwhen that task is gone, as theInputReaderdocs suggest, because a step waits for a disconnected endpoint's answer. Theis_closeddocs now say so.disconnect_inputandextendedrequest that step.remove_input_endpoint.Describe Manual Test Plan
New tests, each checked to fail under the change named:
input_disconnected_before_journaling_replaysdisconnect_inputremoves at once, as before ("unknown input endpoint name 'http'")input_reported_after_disconnect_replaysQueueGatedEndpoint) answers the step'sQueueonly afterdisconnect_input; restartsinput_collected_while_stopping_replaysstopremoves a new endpoint after its step collected input, before the journal write; restartspending_new_inputs("unknown input endpoint name 'late'")input_disconnected_while_paused_is_removed_without_stepinput_disconnected_before_checkpoint_is_not_restoreddisconnected_input_is_not_restartedendpoint_is_closed_after_its_task_dies(dbsp_connectorslib)HttpInputEndpoint::is_closedreturnsfalseremoved_input_endpoint_input_is_still_countednow waits until the endpoint is gone before its late report, so it always covers the size-only credit inorphaned_input; without that credit it fails. The existing W1 and #7672 tests now hold the step until theINSERThas disconnected its endpoint, not until the endpoint is gone; the #7672 test fails ifdisconnect_inputremoves at once.The controller suite (
cargo test -p static-connectors --test connector_suites controller::) passes 98 of 98 in 9 of 10 runs. The other run failed inreplay_does_not_feed_http_input_during_commit, from a deadlock in theREPLAY_GATEStest hook that main has too (it calls the gate with the gates lock held); I will fix that separately.cargo test -p dbsp_connectors --libpasses except the same 26 Kafka, Postgres and DynamoDB tests that fail without this change here (they need those services).Checklist