Repository navigation
Report the latest step request that the circuit saw - #7922
Conversation
ca7d31a to
ef8f33e
Compare
| /// The [StepRequest::seq] of the last step request that the pipeline acted | ||
| /// on. The pipeline acts on a request when it decides from it whether to | ||
| /// start a step, so a status with this sequence number already reflects | ||
| /// that decision. | ||
| /// | ||
| /// The coordinator uses this to tell a pipeline that saw a request to stay | ||
| /// idle from one that has not seen it yet and may still start a step. |
There was a problem hiding this comment.
pitfall (medium): This contract says more than the host guarantees. For a Trigger request at step N with too little buffered input, trigger() returns Park, so the host reports Idle(N) with request_seq = S. When input arrives, the host starts step N under the same S. The stalled branch behaves the same way: it reports S for a Step request and then steps under S once buffer space frees up. So request_seq == S on an Idle status only means that the host will never again decide from a request older than S. It does not mean the host decided to stay idle for S. A coordinator that relies on the current wording could read a host as "won't step" and deadlock the same way. Please state the weaker guarantee here: the coordinator has to bump seq when it cancels and then wait for request_seq >= new seq or Step(N).
There was a problem hiding this comment.
Agreed, the wording promised too much. After the host reports the number of a Trigger request, it can still start the step from it, and the stalled-output branch is the same. Fixed in 54d3df7. The doc now states the weaker guarantee: once the host reports a number, it does not start a step from an older request. Only an Idle request keeps it from starting the step. The controller comments say the same.
The coordinator already depended only on this: cancel_step sends an Idle request with a new number and waits for the host to report that number or the step. It checks for equality, not >=. While it waits, it sends no newer request, so the numbers do not need an order.
|
Review: host-side
Suggested test (crates/connectors/src/controller/test.rs)+/// The step status reports the sequence number of the last step request that
+/// the circuit thread acted on, whether or not it started a step.
+#[engine_test]
+pub fn multihost_step_status_reports_request_seq(engine: &dyn PipelineTestEngine) {
+ use feldera_types::coordination::{StepAction, StepInputs, StepRequest};
+
+ init_test_logger();
+ let config: PipelineConfig = serde_json::from_value(json!({
+ "name": "test", "workers": 1, "clock_resolution_usecs": null,
+ "multihost": { "hosts": 1 },
+ }))
+ .unwrap();
+ let controller = engine
+ .controller(&TestCircuit::identity(relation()), &config, Box::new(|e, _| panic!("error: {e}")))
+ .unwrap();
+ controller.start();
+ let watcher = controller.step_watcher();
+ assert_eq!(watcher.borrow().request_seq, None);
+ let step = watcher.borrow().step;
+
+ controller.set_coordination_request(StepRequest::new_idle(step).with_seq(1));
+ wait(|| watcher.borrow().request_seq == Some(1), 10_000).expect("seq 1 never reported");
+ assert_eq!((watcher.borrow().step, watcher.borrow().action), (step, StepAction::Idle));
+
+ controller.set_coordination_request(
+ StepRequest::new(step, StepAction::Step, StepInputs::None).with_seq(2),
+ );
+ wait(|| { let s = *watcher.borrow(); s.step > step && s.action == StepAction::Idle }, 10_000)
+ .expect("step never ran");
+ assert_eq!(watcher.borrow().request_seq, Some(2));
+
+ let after = watcher.borrow().step;
+ controller.set_coordination_request(StepRequest::new_idle(after).with_seq(3));
+ wait(|| watcher.borrow().request_seq == Some(3), 10_000).expect("seq 3 never reported");
+ assert_eq!(watcher.borrow().step, after);
+ controller.stop().unwrap();
+}I found no other gate failures. The new fields are |
|
Thanks. In 54d3df7:
The controller tests that drive step requests pass, the coordinator tests in feldera/cloud#2205 pass against this branch, and clippy is clean. |
54d3df7 to
7930601
Compare
mihaibudiu
left a comment
There was a problem hiding this comment.
do these parties trust each other to follow the protocol?
what happens if the coordinator sends a smaller sequence number?
I didn't notice any logging for unexpected values.
If they don't, it won't work, but it's not "unsafe" in a Rust sense.
The pipeline just echoes back what the coordinator sends it. The pipeline doesn't interpret that number.
I guess we could log something if the value isn't strictly monotonically increasing. |
A multihost host reports `Idle(N)` both before the coordinator triggers step N and after it cancels it. So when the coordinator cancels a step, `Idle(N)` does not tell it whether the host saw the cancellation. A host whose circuit thread already decided from the trigger to start the step can still report `Step(N)` afterward. The coordinator then thinks the step was cancelled, and that host waits forever for the other hosts to join the step (feldera/cloud#2190). Now the coordinator numbers its step requests with `StepRequest::seq`. The circuit thread reports the number of the latest request that it saw in `StepStatus::request_seq`. Once it reports a number, it does not start a step from an older request. It can still start a step from a `Trigger` whose number it reported, but not from an `Idle` request. So to cancel a step, the coordinator waits for the number of its `Idle` request. The status has no number until the host sees a numbered request. The new `step_request_seq` capability tells the coordinator that the host reports numbers, so that it can tell a host that has not seen a numbered request from an older host. The host does not interpret the numbers, but it logs a warning if the coordinator sends a smaller number than before, or the same number on a different request. A retry of the same request is OK. Adds a controller test of the numbers that the host reports, and serde tests for messages with and without the new fields. Signed-off-by: Ben Pfaff <blp@feldera.com>
7930601 to
36280fb
Compare
|
Added in 36280fb (amended into the single commit). The host still does not interpret -- Claude Code, replying at @blp's request. |
Host side of a fix for a multihost deadlock: a host could start a step that the coordinator had cancelled, and then wait forever for the other hosts to join it.
Problem
A host reports
Idle(N)both before the coordinator triggers step N and after the coordinator cancels it. When the coordinator cancels a step,Idle(N)does not tell it whether the host saw the cancellation. Input can arrive just before the cancellation, so that the circuit thread decides from the trigger to start the step but reportsStep(N)only later. The coordinator then takes the step as cancelled. The host runs step N alone, and its circuit thread stops processing commands, for example checkpoint prepare and checkpoint push.Change
StepRequest::seq: the coordinator numbers each round of step requests.StepStatus::request_seq: the circuit thread reports the number of the latest request that it saw. Once it reports a number, it does not start a step from an older request. It can still start a step from aTriggerwhose number it reported, so to cancel a step the coordinator waits for the number of itsIdlerequest. The host has no number until it sees a numbered request.CoordinationCapabilities::step_request_seq: tells the coordinator that the host reports numbers. Then the coordinator can tell a host that has not yet seen a numbered request from an older host.The new fields are optional in both directions, so old and new hosts and coordinators work together. With an older coordinator, the host reports no number.
The coordinator side is a separate PR in the cloud repo. It needs this PR first.
Testing
multihost_step_status_reports_request_seq(controller) checks the numbers the host reports forIdle,Trigger, andSteprequests, including a step started from aTriggerwhose number the host already reported. It fails without the report in the main loop.step_request_seq_is_optionalandstep_request_seq_round_trips(feldera-types) check messages with and without the new fields.step_status_reports_committed_recordsandmultihost_api_transaction_committed_before_it_starts. The multihost, coordination, and checkpoint connector suites also pass, except Kafka and Postgres tests that need services I did not run locally.dbsp_connectorsandfeldera-types.