← 输入框|github.com|刷新
Skip to content

Report the latest step request that the circuit saw - #7922

Merged
blp merged 1 commit into
mainfrom
step-request-seq
Nov 11, 2026
Merged

blp merged 1 commit into
mainfrom
step-request-seq

Conversation

@blp

@blp blp commented Nov 10, 2026 •

Copy link
Copy Markdown
Member

Host side of a fix for a multihost deadlock: a host could start a step that the coordinator had cancelled, and then wait forever for the other hosts to join it.

Problem

A host reports Idle(N) both before the coordinator triggers step N and after the coordinator cancels it. When the coordinator cancels a step, Idle(N) does not tell it whether the host saw the cancellation. Input can arrive just before the cancellation, so that the circuit thread decides from the trigger to start the step but reports Step(N) only later. The coordinator then takes the step as cancelled. The host runs step N alone, and its circuit thread stops processing commands, for example checkpoint prepare and checkpoint push.

Change

  • StepRequest::seq: the coordinator numbers each round of step requests.
  • StepStatus::request_seq: the circuit thread reports the number of the latest request that it saw. Once it reports a number, it does not start a step from an older request. It can still start a step from a Trigger whose number it reported, so to cancel a step the coordinator waits for the number of its Idle request. The host has no number until it sees a numbered request.
  • CoordinationCapabilities::step_request_seq: tells the coordinator that the host reports numbers. Then the coordinator can tell a host that has not yet seen a numbered request from an older host.

The new fields are optional in both directions, so old and new hosts and coordinators work together. With an older coordinator, the host reports no number.

The coordinator side is a separate PR in the cloud repo. It needs this PR first.

Testing

  • multihost_step_status_reports_request_seq (controller) checks the numbers the host reports for Idle, Trigger, and Step requests, including a step started from a Trigger whose number the host already reported. It fails without the report in the main loop.
  • step_request_seq_is_optional and step_request_seq_round_trips (feldera-types) check messages with and without the new fields.
  • The coordinator PR adds the regression test for the race. Its fake hosts copy what this PR does on the host side. The test fails without the fix.
  • The controller tests that drive step requests pass: step_status_reports_committed_records and multihost_api_transaction_committed_before_it_starts. The multihost, coordination, and checkpoint connector suites also pass, except Kafka and Postgres tests that need services I did not run locally.
  • Clippy is clean on dbsp_connectors and feldera-types.

@blp blp added bug Something isn't working connectors Issues related to the adapters/connectors crate multihost Related to multihost or distributed pipelines rust Pull requests that update Rust code labels Nov 10, 2026
@blp blp self-assigned this Nov 10, 2026
@blp
blp force-pushed the step-request-seq branch from ca7d31a to ef8f33e Compare November 10, 2026 20:24
@blp
blp requested a review from gz November 10, 2026 20:24
Comment on lines +429 to +435
/// The [StepRequest::seq] of the last step request that the pipeline acted
/// on. The pipeline acts on a request when it decides from it whether to
/// start a step, so a status with this sequence number already reflects
/// that decision.
///
/// The coordinator uses this to tell a pipeline that saw a request to stay
/// idle from one that has not seen it yet and may still start a step.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pitfall (medium): This contract says more than the host guarantees. For a Trigger request at step N with too little buffered input, trigger() returns Park, so the host reports Idle(N) with request_seq = S. When input arrives, the host starts step N under the same S. The stalled branch behaves the same way: it reports S for a Step request and then steps under S once buffer space frees up. So request_seq == S on an Idle status only means that the host will never again decide from a request older than S. It does not mean the host decided to stay idle for S. A coordinator that relies on the current wording could read a host as "won't step" and deadlock the same way. Please state the weaker guarantee here: the coordinator has to bump seq when it cancels and then wait for request_seq >= new seq or Step(N).

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, the wording promised too much. After the host reports the number of a Trigger request, it can still start the step from it, and the stalled-output branch is the same. Fixed in 54d3df7. The doc now states the weaker guarantee: once the host reports a number, it does not start a step from an older request. Only an Idle request keeps it from starting the step. The controller comments say the same.

The coordinator already depended only on this: cancel_step sends an Idle request with a new number and waits for the host to report that number or the step. It checks for equality, not >=. While it waits, it sends no newer request, so the numbers do not need an order.

@mythical-fred-oss

Copy link
Copy Markdown
Contributor

Review: host-side request_seq reporting

Check Result
cargo check -p feldera-types, cargo test -p feldera-types coordination ✅
static-connectors connector_suites: step_status_reports_committed_records, multihost_api_transaction_committed_before_it_starts, all multihost/coordinat cases ✅
My new test below (on the static engine) ✅
  • missing (high), Unit/Integration test gate: This PR changes host behaviour, but this repo gets no test for it. The regression test lives in another PR, and the existing tests never set seq. Please add a test like the one below. It covers four cases: a request without seq reports None, a numbered idle request is reported without starting a step, a step request's seq appears on the Idle that follows the step, and a numbered cancellation of an older step does not start a step.
  • missing (medium): Nothing tests the stalled-output path, the Synchronize path, or mixing old and new coordinators and hosts. The mixed case is serde round-trips of StepRequest/StepStatus/CoordinationCapabilities with the new fields missing.
Suggested test (crates/connectors/src/controller/test.rs)
+/// The step status reports the sequence number of the last step request that
+/// the circuit thread acted on, whether or not it started a step.
+#[engine_test]
+pub fn multihost_step_status_reports_request_seq(engine: &dyn PipelineTestEngine) {
+    use feldera_types::coordination::{StepAction, StepInputs, StepRequest};
+
+    init_test_logger();
+    let config: PipelineConfig = serde_json::from_value(json!({
+        "name": "test", "workers": 1, "clock_resolution_usecs": null,
+        "multihost": { "hosts": 1 },
+    }))
+    .unwrap();
+    let controller = engine
+        .controller(&TestCircuit::identity(relation()), &config, Box::new(|e, _| panic!("error: {e}")))
+        .unwrap();
+    controller.start();
+    let watcher = controller.step_watcher();
+    assert_eq!(watcher.borrow().request_seq, None);
+    let step = watcher.borrow().step;
+
+    controller.set_coordination_request(StepRequest::new_idle(step).with_seq(1));
+    wait(|| watcher.borrow().request_seq == Some(1), 10_000).expect("seq 1 never reported");
+    assert_eq!((watcher.borrow().step, watcher.borrow().action), (step, StepAction::Idle));
+
+    controller.set_coordination_request(
+        StepRequest::new(step, StepAction::Step, StepInputs::None).with_seq(2),
+    );
+    wait(|| { let s = *watcher.borrow(); s.step > step && s.action == StepAction::Idle }, 10_000)
+        .expect("step never ran");
+    assert_eq!(watcher.borrow().request_seq, Some(2));
+
+    let after = watcher.borrow().step;
+    controller.set_coordination_request(StepRequest::new_idle(after).with_seq(3));
+    wait(|| watcher.borrow().request_seq == Some(3), 10_000).expect("seq 3 never reported");
+    assert_eq!(watcher.borrow().step, after);
+    controller.stop().unwrap();
+}

I found no other gate failures. The new fields are serde(default), so old and new hosts and coordinators still work together. The coordination API is internal, so it needs no user docs. There is no unsafe code and no new dependency.

@blp

blp commented Nov 10, 2026

Copy link
Copy Markdown
Member Author

Thanks. In 54d3df7:

Item Change
Doc wording (inline) StepStatus::request_seq now states the weaker guarantee; see the inline reply.
No host test (high) Added multihost_step_status_reports_request_seq, based on your suggestion. It also checks the weaker guarantee: the host reports a Trigger's number while it has no reason to step, then starts the step from the same Trigger after request_step(). Without the report in the main loop, the test fails with "seq 1 never reported".
Serde with missing fields (medium) Added step_request_seq_is_optional and step_request_seq_round_trips in feldera-types. They check that messages without the new fields read as None/false, and that None fields are left out when serializing.
Stalled-output and Synchronize paths (medium) Not added. Each needs a lot of setup (full output buffers, a concurrent bootstrap) to test one line, and neither path can start a step from an older request.

The controller tests that drive step requests pass, the coordinator tests in feldera/cloud#2205 pass against this branch, and clippy is clean.

@blp
blp force-pushed the step-request-seq branch from 54d3df7 to 7930601 Compare November 10, 2026 22:06
@blp blp changed the title Report which step request the circuit acted on Report the latest step request that the circuit saw Nov 10, 2026

@mihaibudiu mihaibudiu left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do these parties trust each other to follow the protocol?
what happens if the coordinator sends a smaller sequence number?
I didn't notice any logging for unexpected values.

@blp

blp commented Nov 11, 2026

Copy link
Copy Markdown
Member Author

do these parties trust each other to follow the protocol?

If they don't, it won't work, but it's not "unsafe" in a Rust sense.

what happens if the coordinator sends a smaller sequence number?

The pipeline just echoes back what the coordinator sends it. The pipeline doesn't interpret that number.

I didn't notice any logging for unexpected values.

I guess we could log something if the value isn't strictly monotonically increasing.

A multihost host reports `Idle(N)` both before the coordinator triggers
step N and after it cancels it.  So when the coordinator cancels a step,
`Idle(N)` does not tell it whether the host saw the cancellation.  A host
whose circuit thread already decided from the trigger to start the step
can still report `Step(N)` afterward.  The coordinator then thinks the
step was cancelled, and that host waits forever for the other hosts to
join the step (feldera/cloud#2190).

Now the coordinator numbers its step requests with `StepRequest::seq`.
The circuit thread reports the number of the latest request that it saw
in `StepStatus::request_seq`.  Once it reports a number, it does not
start a step from an older request.  It can still start a step from a
`Trigger` whose number it reported, but not from an `Idle` request.  So
to cancel a step, the coordinator waits for the number of its `Idle`
request.  The status has no number until the host sees a numbered
request.  The new `step_request_seq` capability tells the coordinator
that the host reports numbers, so that it can tell a host that has not
seen a numbered request from an older host.

The host does not interpret the numbers, but it logs a warning if the
coordinator sends a smaller number than before, or the same number on a
different request.  A retry of the same request is OK.

Adds a controller test of the numbers that the host reports, and serde
tests for messages with and without the new fields.

Signed-off-by: Ben Pfaff <blp@feldera.com>
@blp
blp force-pushed the step-request-seq branch from 7930601 to 36280fb Compare November 11, 2026 01:30
@blp

blp commented Nov 11, 2026

Copy link
Copy Markdown
Member Author

Added in 36280fb (amended into the single commit). The host still does not interpret seq, but set_coordination_request() now logs a warning when the coordinator sends a smaller number than the last numbered request, or the same number on a different request. The same request sent again with the same number is a retry, so it gives no warning. The coordinator puts a number only on cancellation requests and starts from 0 after a restart. A restart does not trigger the warning, because the new coordinator activates the pipelines again. The new unit test unexpected_step_request_seq fails if the check is disabled.

-- Claude Code, replying at @blp's request.

@blp
blp enabled auto-merge November 11, 2026 01:33
@blp
blp disabled auto-merge November 11, 2026 01:50
@blp
blp added this pull request to the merge queue Nov 11, 2026
Merged via the queue into main with commit c51c29d Nov 11, 2026
1 check passed
@blp
blp deleted the step-request-seq branch November 11, 2026 04:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working connectors Issues related to the adapters/connectors crate multihost Related to multihost or distributed pipelines rust Pull requests that update Rust code

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants