Repository navigation
[connectors] Make pause take effect more promptly. - #7941
Conversation
|
@ryzhyk This is a draft only because I am uncertain whether we want to fix this behavior at all, since it is so long-standing. On the other hand, the behavior is always confusing. |
|
Sounds like a useful feature |
| let state = match global_metrics.get_state() { | ||
| PipelineState::Terminated => PipelineState::Terminated, | ||
| _ => global_metrics.reported_state(), | ||
| }; |
There was a problem hiding this comment.
bug (medium): /status now reports reported_state, but /stats (ExternalGlobalControllerMetrics.state, controller/stats.rs ~L1706) still serializes the requested state. A client that polls pipeline.stats().global_metrics.state == PAUSED sees Paused while a step can still take connector input, so it does not get the guarantee this PR adds, and the two endpoints disagree for up to one step. Suggest serializing reported_state() there too, except for Terminated as here.
There was a problem hiding this comment.
Fixed. GlobalControllerMetrics::reported_state() now returns Terminated once the pipeline terminates, and /stats serializes it, as /status does. The two endpoints now use the same rule.
-- Claude Code, replying at @blp's request.
| A paused pipeline does not process the input that its connectors | ||
| buffered, except forced input to HTTP input connectors. When this | ||
| function waits, it returns after the step that was in progress at the | ||
| time of the pause is complete. |
There was a problem hiding this comment.
pitfall (medium): wait_for_idle() returns only when total_input_records == total_processed_records. A paused pipeline now holds buffered input, and the PR says connectors can still buffer for a short time after the pause, so pause(); wait_for_idle() (the pattern in tests/workloads/test_now.py) can now hang until its timeout. Please say this here and in the wait_for_idle docstring and testing.md, or have wait_for_idle compare total_processed + buffered while paused.
There was a problem hiding this comment.
Documented in the wait_for_idle docstring and in testing.md. I did not change wait_for_idle to count buffered input while paused: then it would also return early while the pipeline runs with input that is not yet processed, unless it first checked the status, and "idle" would mean different things in the two states.
tests/workloads/test_now.py is not affected: it waits for buffered_input_records == 0 before it pauses, and its datagen input is finite.
-- Claude Code, replying at @blp's request.
There was a problem hiding this comment.
Update: on reflection, I changed wait_for_idle after all. It now reads state and buffered_input_records from the same /stats response. For a paused pipeline, it counts buffered input as processed (input == processed + buffered), and it resets the idle interval when the buffered count changes. A running pipeline uses the old rule. /stats now reports the same state as /status, so a reported pause means that no step takes that input. New unit tests: python/tests/unit/test_wait_for_idle.py. The paused case fails with the old rule.
-- Claude Code, replying at @blp's request.
|
What I ran (debug,
missing (medium): forced HTTP input while paused. Only + /// A paused pipeline still processes forced HTTP input, and `/status`
+ /// reports `Paused` throughout.
+ #[engine_test]
+ pub fn paused_pipeline_processes_forced_http_input(engine: &dyn PipelineTestEngine) {
+ actix_web::rt::System::new().block_on(async {
+ crate::ensure_default_crypto_provider();
+ let server = start_test_server(
+ engine,
+ &json!({"name": "test", "inputs": {}, "outputs": {}}),
+ Uuid::new_v4(),
+ )
+ .await;
+ start_pipeline(&server).await;
+ send_input(&server, &test_batches(0, 10)).await;
+ pause_pipeline(&server).await;
+ let CompletionTokenResponse { token } =
+ TestHttpSender::send_stream_deserialize_resp::<CompletionTokenResponse>(
+ server.post("/ingress/test_input1?force=true"),
+ &test_batches(10, 10),
+ )
+ .await;
+ wait_for_completion(&server, &token).await;
+ assert_eq!(get_stats(&server).await.global_metrics.total_processed_records, 20);
+ wait_for_status(&server, RuntimeStatus::Paused).await;
+ })
+ }Other uncovered cases (medium):
None of these has a test. Gates: tests, flakiness, docs and the breaking-change changelog entry all pass. The two inline comments cover the |
fb82f18 to
365aa7f
Compare
42413b8 to
2ac9938
Compare
|
Thanks. Changes for the uncovered cases:
The -- Claude Code, replying at @blp's request. |
2ac9938 to
bcebd9f
Compare
|
Update on the uncovered cases:
The full local connector run, with Postgres, Redis, DynamoDB and Redpanda, is still in progress. -- Claude Code, replying at @blp's request. |
|
End-to-end check on a local k3d cluster. A pipeline has a datagen table (200 rows/s) and an HTTP-only table. The script starts it, pauses, waits 5 s, sends 10 rows with
-- Claude Code, replying at @blp's request. |
91c9437 to
3a86863
Compare
3a86863 to
6df3c62
Compare
4805223 to
3aafb87
Compare
6df3c62 to
9300897
Compare
| | Accepts input from the SDK only when the input is forced | `pipeline.input_json()` raises a `RuntimeError` unless the call passes `force=True`. | | ||
| | Does not advance the clock | While the pipeline is paused, the clock connector does not initiate new steps, as it does normally. The value returned by `NOW()` changes in steps initiated by another reason, for example to process buffered or forced input. Such a step *may* advance the clock: either reading the system clock, or, (when `now_http_driven` is configured) to the value supplied by the last `advance_clock()` call executed. | | ||
| | Accepts input from the SDK only when the input is forced | `pipeline.input_json()` raises a `RuntimeError` unless the call passes `force=True`. The pipeline processes forced input, and any other input that the HTTP input connectors buffered. | | ||
| | Does not advance the clock | The clock connector does not initiate new steps, and the pipeline does not take new clock values from it. `NOW()` keeps its value until `resume()`. | |
There was a problem hiding this comment.
pitfall (medium): With now_http_driven, advance_clock() while paused still returns the new now_ms and requests a step, but that step skips the clock endpoint. NOW() doesn't move until resume(). The old row documented the opposite, and the Pipeline.advance_clock() docstring still says the value is emitted "on its next pipeline step" and suggests polling the view. Polling would hang while paused. Please update that docstring and spell out the advance_clock() case here. I confirmed this with a scratch test:
pipeline.pause();
wait(|| status.global_metrics.reported_state() == PipelineState::Paused, 10_000).unwrap();
let steps = status.global_metrics.total_completed_steps();
assert_eq!(advance(&pipeline, Some(60_000)).unwrap(), millis(HTTP_ANCHOR) + 60_000);
wait(|| status.global_metrics.total_completed_steps() > steps, 10_000).unwrap();
assert_eq!(ticks(&pipeline).and_then(|(_, l)| l), Some(millis(HTTP_ANCHOR))); // unchanged
pipeline.start();
wait_for_clock_timestamp(&pipeline, millis(HTTP_ANCHOR) + 60_000); // applied on resumeIt passes. Consider adding it to transport/clock.rs as a regression test for the new contract.
There was a problem hiding this comment.
Done, in bd09a8304:
- New regression test
advance_http_driven_while_pausedintransport/clock.rs, based on your scratch test. It fails if a paused step polls the clock endpoint. - The
advance_clock()docstring now says that a paused pipeline takes the new value only afterresume(), and that polling the view for it must wait until then. - The clock row in
testing.mdnow coversadvance_clock()withnow_http_driven.
-- Claude Code, replying at @blp's request.
|
What I ran
Gates: tests, docs and changelog are present. The breaking change is called out. I found no flakiness: the new tests wait on step counters, not sleeps. No new dependencies and no Not covered by tests:
|
9300897 to
bd09a83
Compare
|
Thanks. Changes in
Python unit tests (436), the clock tests and pre-commit pass. -- Claude Code, replying at @blp's request. |
7cdd61a to
573faaf
Compare
|
Most questions are around what the pipeline can still do in a PAUSED state. This is the only actual bug I found: We can still be committing a transaction in the PAUSED state: Not a regression. Claude considers it a bug that we can freeze the pipeline mid-transaction. This doesn't make things worse than today. It's odd though that we don't pause in the middle of an API-triggered transaction, but we do pause in the middle of a connector-driver transaction. |
ryzhyk
left a comment
There was a problem hiding this comment.
It's unclear how /pause should interact with transactions. We have two options: (1) preserve the current behavior by addressing the first issue I reported. This is the minimal fix that will make the PR acceptable. (2) figure out a more reasonable/consistent behavior and implement that. I'm ok either way.
bd09a83 to
fbe69e1
Compare
|
Thanks. This takes option 1, with the first fix from issue 1. Changes in
-- Claude Code, replying at @blp's request. |
ryzhyk
left a comment
There was a problem hiding this comment.
Thanks for the fixes!
Claude recommends adjusting wording in docs in several places to match the actual behavior:
Doc and current wording: testing.md, clock row: "The clock connector does not initiate new steps, and the pipeline does not take new clock values from it. NOW() keeps its value until resume(). With now_http_driven, advance_clock()
returns the new value, but the pipeline takes it only after resume()."
Verdict: Broken
Evidence: Probe 7: NOW() moved to the advanced value while /status said Paused.
Proposed wording: "The clock connector does not initiate new steps, and NOW() keeps its value until resume(), with one exception: from a commit_transaction() request until the commit starts, the pipeline takes a new clock value
(see below). With now_http_driven, advance_clock() returns the new value at once. The pipeline takes it after resume() or during that exception, whichever comes first."
────────────────────────────────────────
Doc and current wording: SDK Pipeline.advance_clock() docstring: "A paused pipeline does not take the new value until it resumes, so while the pipeline is paused, the views keep the previous NOW(). Do not poll the view for the new
value until after :meth:.resume."
Verdict: Broken
Evidence: Probe 7
Proposed wording: "A paused pipeline takes the new value when it resumes, or earlier if it commits a transaction: from a :meth:.commit_transaction request until the commit starts, it polls all its connectors, the clock included.
Otherwise, while the pipeline is paused, the views keep the previous NOW(), so do not poll the view for the new value until after :meth:.resume."
────────────────────────────────────────
Doc and current wording: testing.md, "Transaction commit" row: "A transaction that commits while the pipeline is paused includes the input that the connectors buffered before the commit request, so the pipeline processes that input
to finish the commit. A commit that was in progress at the pause also finishes."
Verdict: Understated
Evidence: Probe 8: rows queued after the request went in while paused, and the commit was still blocked on a joined connector. Repro 2: a transaction that a connector started stayed open, 0 of 95 rows committed.
Proposed wording: "From a commit_transaction() request until the commit starts, a paused pipeline polls all its connectors, the clock included, as a running pipeline does. The transaction thus includes the input that the connectors
buffered before the request, and any input that arrives before the commit starts. If a connector in the transaction has not committed its part, this lasts until it does, which can be after resume(). A commit that was in progress
at the pause also finishes. A transaction that only connectors started does not commit while the pipeline is paused, unless its commit started before the pause. Until it commits, the pipeline takes no checkpoints, and a suspend
waits."
────────────────────────────────────────
Doc and current wording: testing.md, paragraph above the table: "A connector can continue to buffer input for a short time after the pause, but the pipeline does not process that input until resume()."
Verdict: Understated (same cause as the row above)
Evidence: Probe 8
Proposed wording: "A connector can continue to buffer input for a short time after the pause. Except in the cases in the table below, the pipeline does not process that input until resume()."
────────────────────────────────────────
Doc and current wording: testing.md, below the table: "A multihost pipeline behaves the same way."
Verdict: Understated (inferred)
Evidence: ControllerInner::start_commit_transaction_from_api sets commit_target only for a single-host pipeline: "the coordinator commits with steps that feed no input". I did not check the coordinator.
Proposed wording: "A multihost pipeline behaves the same way, except that a transaction commit does not take the input that the connectors buffered."
────────────────────────────────────────
Doc and current wording: lifecycle.md, Paused row: "The input connectors are paused, and the pipeline does not process the input that they buffered. Only forced input on HTTP input connectors goes in."
Verdict: Stale
Evidence: Lists no exceptions. Non-forced HTTP input buffered before the pause also goes in.
Proposed wording: "The input connectors are paused, and the pipeline does not process the input that they buffered. The exception is input in the HTTP input connectors, which accept new input while paused only with force=true. For
other exceptions, such as a transaction commit or a checkpoint, see Pausing a pipeline (/tutorials/testing#pausing-a-pipeline)."
────────────────────────────────────────
Doc and current wording: SDK Pipeline.pause() docstring: "A paused pipeline does not process the input that its connectors buffered, except forced input to HTTP input connectors. When this function waits, it returns after the step
that was in progress at the time of the pause is complete."
Verdict: Stale
Evidence: Lists no exception besides HTTP. Repro 3: a commit running at the pause changes the views after pause() returns.
Proposed wording: "A paused pipeline does not process the input that its connectors buffered. The exceptions are input in HTTP input connectors, such as forced input from :meth:.input_json, and the cases listed under "Pausing a
pipeline" in the testing guide, such as a transaction commit. When this function waits, it returns after the step that was in progress at the time of the pause is complete. A transaction commit that was in progress at the pause
still finishes after that, which changes the views."
────────────────────────────────────────
Doc and current wording: PR description: "A reported pause thus means that no step takes connector input." Its exceptions list has has_run, checkpoint and coordinator.
Verdict: Stale
Evidence: The exceptions list lacks the commit exception from b29394dbd.
Proposed wording: Sentence: "A reported pause thus means that no step takes input from connectors other than HTTP input endpoints, except in the cases below." New bullet: "From an API commit_transaction request until the commit
starts, steps poll all endpoints, so that the commit includes the input queued at the request. This lasts until every connector in the transaction has committed its part. A commit in progress at the pause also finishes."
|
Thanks.
The PR description now has the commit exception, and its pause sentence allows HTTP input and the listed cases. -- Claude Code, replying at @blp's request. |
Before, a paused single-host pipeline continued to poll its input connectors, so input that they buffered before or shortly after the pause still went into the circuit. No API could confirm that input had stopped, which made tests nondeterministic. Now, without a coordinator, a paused pipeline steps with `StepInputs::Paused`, which polls only HTTP input endpoints (they accept input while paused only with `force=true`), and during a checkpoint, with `StepInputs::CheckpointBarriers` as before. The circuit thread records the state that it applied, and the pipeline reports `Paused` only after it applies the pause, so no step in progress can take more input. A paused pipeline that has never run still polls every endpoint once, so that each connector records its starting position for checkpoints. With feldera/cloud#2209, a multihost pipeline behaves the same way. For a paused pipeline, the Python SDK's `wait_for_idle()` counts the input that connectors hold in buffers as if it were processed, because the pipeline does not process it until it resumes. The exception is input in HTTP input connectors, which a paused pipeline still processes. With `now_http_driven`, `advance_clock()` while paused returns the new `NOW()`, but the pipeline takes it only after it resumes. Fixes #7568. Signed-off-by: Ben Pfaff <blp@feldera.com>
An API commit includes the input that connectors queued before the commit request, so it waits until a step polls every input endpoint. A paused pipeline polled only HTTP input endpoints, so the commit never finished, and the circuit thread kept stepping to advance it. Now, while an API commit waits for queued input, a paused pipeline polls all its input endpoints. When the commit starts, the pause applies again. The docs now list a transaction commit as a case in which a paused pipeline processes buffered input. Signed-off-by: Ben Pfaff <blp@feldera.com>
From a `commit_transaction()` request until the commit starts, a paused pipeline polls all its connectors, the clock included. The docs for the clock, `advance_clock()`, `pause()` and the Paused state now say so, and also that a transaction that only connectors started does not commit while the pipeline is paused. A multihost commit does not take buffered input. Signed-off-by: Ben Pfaff <blp@feldera.com>
v0.383.0 was released before this change merged. Signed-off-by: Ben Pfaff <blp@feldera.com>
26a2fa9 to
0db9bfe
Compare
Fixes #7568. Uses
StepInputs::Pausedfrom #7974, which has merged.A paused single-host pipeline continued to poll its input connectors. Input that connectors buffered before the pause, or shortly after it, still went into the circuit, and no API could confirm that input had stopped. This made tests nondeterministic.
Change
force=true. Other buffered input waits until resume.Pausedat once, while a step can still take input.reported_statebetween steps. Steps select their input byreported_state, and the server reports it. A reported pause thus means that no step takes input from connectors other than HTTP input endpoints, except in the cases below.The paused selection is
StepInputs::Paused, from #7974. A coordinator requests the same selection for a paused multihost pipeline (feldera/cloud#2209).Exceptions:
has_run), steps poll all endpoints. No reader has buffered input yet, and the first poll records each connector's resume position, which a checkpoint needs.commit_transactionrequest until the commit starts, steps poll all endpoints, so that the commit includes the input queued at the request. This lasts until every connector in the transaction has committed its part. A commit in progress at the pause also finishes.StepInputs::Pausedwhile paused.This is an incompatible change. The docs (
testing.md,lifecycle.md,transactions.md, changelog) and the docstrings of the SDK'spause()andadvance_clock()describe the new behavior.Testing
paused_pipeline_does_not_poll_buffered_input,paused_pipeline_polls_until_it_first_runs,paused_pipeline_processes_forced_http_input,paused_suspend_polls_barrier_inputandpaused_api_transaction_commit_includes_queued_records. Each fails without its change./statsreports the same state as/status.connector_suitesforcontroller::,server::,transport::httpanddelta_table: all 350 pass, with the test services (Postgres, Redis, DynamoDB, Redpanda) running.