Repository navigation
[types,connectors] Add StepInputs::Paused for a paused multihost pipeline. - #7974
Conversation
|
LGTM. The change is small and correct. A locally paused controller still runs coordinator steps, because pause only drives backpressure, and a paused HTTP reader still flushes on What I ran
Missing (medium): the test does not cover the paused case it is named after. diff/// Like `multihost_paused_step_polls_only_http_input`, but with the pipeline
/// actually paused and the HTTP input forced, as in production.
#[engine_test]
pub fn multihost_paused_step_takes_forced_http_input(engine: &dyn PipelineTestEngine) {
use feldera_types::coordination::{StepAction, StepInputs, StepRequest};
init_test_logger();
let config: PipelineConfig = serde_json::from_value(json!({
"name": "test", "workers": 4, "clock_resolution_usecs": null,
"multihost": { "hosts": 1 },
})).unwrap();
let controller = engine
.controller(&identity(&[None]), &config, Box::new(|e, _| panic!("error: {e}")))
.unwrap();
controller.pause();
let http = HttpInputEndpoint::new(HttpInputConfig { name: "http".into() });
controller
.add_input_endpoint("http", http_input_config("http"), Box::new(http.clone()), None)
.unwrap();
TOKIO.block_on(async {
let (request, mut payload) = actix_web::test::TestRequest::default()
.set_payload(r#"{"id": 1, "b": true, "i": 1, "s": "x"}"#)
.to_http_parts();
let payload = <actix_web::web::Payload as actix_web::FromRequest>::from_request(
&request, &mut payload).await.unwrap();
http.complete_request(payload, true, None).await.unwrap();
});
let step = controller.step_watcher().borrow().step;
controller.set_coordination_request(StepRequest::new(step, StepAction::Step, StepInputs::Paused));
wait(|| controller.step_watcher().borrow().step > step, 10_000)
.expect("the coordinator's step never ran");
let status = controller.status();
wait(|| status.num_total_processed_records() == 1, 10_000)
.expect("the paused step did not take the forced HTTP input");
controller.stop().unwrap();
}No feldera-pr-checks gate fails. Docs are not needed: |
4805223 to
3aafb87
Compare
|
Thanks. Fixed in 3aafb87, squashed into the single commit. I did not add a second, almost identical test. I changed
The test still checks that the other connector's input stays buffered. It fails with -- Claude Code, replying at @blp's request. |
| /// A pipeline without this capability cannot parse a step request with | ||
| /// that value. | ||
| #[serde(default)] | ||
| pub paused_step_inputs: bool, |
There was a problem hiding this comment.
maybe this should be called accepts_paused_step_inputs
|
|
||
| /// The pipeline is paused. Consider input arriving only on HTTP input | ||
| /// endpoints to trigger a step, and for executing a step, use input only | ||
| /// from them. While the pipeline is paused, they accept input only from |
3aafb87 to
7cdd61a
Compare
…peline. A paused multihost pipeline cannot process forced HTTP input, because its coordinator does not start steps while the pipeline is paused. Now a coordinator can request a step with `StepInputs::Paused`. Such a step polls only HTTP input endpoints, which accept input while paused only from requests with `force=true`, and records no `StepFeed`, as for `CheckpointBarriers`. The coordinator change is feldera/cloud#2209, for feldera/cloud#2208. A pipeline reports the new `accepts_paused_step_inputs` capability, because an older pipeline cannot parse a step request with `StepInputs::Paused`. This changes nothing for a single-host pipeline. Signed-off-by: Ben Pfaff <blp@feldera.com>
7cdd61a to
573faaf
Compare
A paused multihost pipeline cannot process forced HTTP input, because its coordinator does not start steps while the pipeline is paused (feldera/cloud#2208).
This PR lets a coordinator request a step with
StepInputs::Paused:StepInputs::Paused(feldera-types)force=true.StepSelection::polls()(crates/connectors/src/controller.rs)CheckpointBarriers, such a step records noStepFeed.CoordinationCapabilities::accepts_paused_step_inputsStepInputs::Paused. The coordinator sends it only if every host reports it.multihost_paused_step_polls_only_http_inputPausedstep takes HTTP input and leaves the other connector's input buffered. It fails ifPausedpolls every endpoint.Nothing changes for a single-host pipeline. The coordinator side is feldera/cloud#2209.
This was split out of #7941, which changes single-host pause behavior and waits for a design decision. #7941 is now stacked on this PR.
Tests: controller, server and HTTP suites (162) pass; clippy is clean.