← 输入框|github.com|刷新
Skip to content

[connectors] Make pause take effect more promptly. - #7941

Merged
blp merged 4 commits into
mainfrom
inputs-quiescent
Nov 17, 2026
Merged

blp merged 4 commits into
mainfrom
inputs-quiescent

Conversation

@blp

@blp blp commented Nov 11, 2026 •

Copy link
Copy Markdown
Member

Fixes #7568. Uses StepInputs::Paused from #7974, which has merged.

A paused single-host pipeline continued to poll its input connectors. Input that connectors buffered before the pause, or shortly after it, still went into the circuit, and no API could confirm that input had stopped. This made tests nondeterministic.

Change

Before After
A paused pipeline processes all buffered input. A paused pipeline polls only HTTP input endpoints. These accept input while paused only with force=true. Other buffered input waits until resume.
The server reports Paused at once, while a step can still take input. The circuit thread copies the state to reported_state between steps. Steps select their input by reported_state, and the server reports it. A reported pause thus means that no step takes input from connectors other than HTTP input endpoints, except in the cases below.

The paused selection is StepInputs::Paused, from #7974. A coordinator requests the same selection for a paused multihost pipeline (feldera/cloud#2209).

Exceptions:

  • Until the pipeline first runs (has_run), steps poll all endpoints. No reader has buffered input yet, and the first poll records each connector's resume position, which a checkpoint needs.
  • To make a checkpoint, a pipeline without fault tolerance still polls the endpoints that block the checkpoint.
  • From an API commit_transaction request until the commit starts, steps poll all endpoints, so that the commit includes the input queued at the request. This lasts until every connector in the transaction has committed its part. A commit in progress at the pause also finishes.
  • With a coordinator, the coordinator decides. The cloud PR for feldera/cloud#2208 makes it step with StepInputs::Paused while paused.

This is an incompatible change. The docs (testing.md, lifecycle.md, transactions.md, changelog) and the docstrings of the SDK's pause() and advance_clock() describe the new behavior.

Testing

  • New tests paused_pipeline_does_not_poll_buffered_input, paused_pipeline_polls_until_it_first_runs, paused_pipeline_processes_forced_http_input, paused_suspend_polls_barrier_input and paused_api_transaction_commit_includes_queued_records. Each fails without its change.
  • /stats reports the same state as /status.
  • connector_suites for controller::, server::, transport::http and delta_table: all 350 pass, with the test services (Postgres, Redis, DynamoDB, Redpanda) running.
  • Not run locally: the Python platform tests.

@blp blp added bug Something isn't working documentation Improvements or additions to documentation connectors Issues related to the adapters/connectors crate User-facing For PRs that lead to Feldera-user visible changes labels Nov 11, 2026
@blp blp self-assigned this Nov 11, 2026
@blp
blp requested a review from ryzhyk November 11, 2026 19:41
@blp

blp commented Nov 11, 2026

Copy link
Copy Markdown
Member Author

@ryzhyk This is a draft only because I am uncertain whether we want to fix this behavior at all, since it is so long-standing. On the other hand, the behavior is always confusing.

@mihaibudiu

Copy link
Copy Markdown
Contributor

Sounds like a useful feature

Comment thread crates/connectors/src/server.rs Outdated
let state = match global_metrics.get_state() {
PipelineState::Terminated => PipelineState::Terminated,
_ => global_metrics.reported_state(),
};

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bug (medium): /status now reports reported_state, but /stats (ExternalGlobalControllerMetrics.state, controller/stats.rs ~L1706) still serializes the requested state. A client that polls pipeline.stats().global_metrics.state == PAUSED sees Paused while a step can still take connector input, so it does not get the guarantee this PR adds, and the two endpoints disagree for up to one step. Suggest serializing reported_state() there too, except for Terminated as here.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed. GlobalControllerMetrics::reported_state() now returns Terminated once the pipeline terminates, and /stats serializes it, as /status does. The two endpoints now use the same rule.

-- Claude Code, replying at @blp's request.

Comment thread python/feldera/pipeline.py Outdated
A paused pipeline does not process the input that its connectors
buffered, except forced input to HTTP input connectors. When this
function waits, it returns after the step that was in progress at the
time of the pause is complete.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pitfall (medium): wait_for_idle() returns only when total_input_records == total_processed_records. A paused pipeline now holds buffered input, and the PR says connectors can still buffer for a short time after the pause, so pause(); wait_for_idle() (the pattern in tests/workloads/test_now.py) can now hang until its timeout. Please say this here and in the wait_for_idle docstring and testing.md, or have wait_for_idle compare total_processed + buffered while paused.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Documented in the wait_for_idle docstring and in testing.md. I did not change wait_for_idle to count buffered input while paused: then it would also return early while the pipeline runs with input that is not yet processed, unless it first checked the status, and "idle" would mean different things in the two states.

tests/workloads/test_now.py is not affected: it waits for buffered_input_records == 0 before it pauses, and its datagen input is finite.

-- Claude Code, replying at @blp's request.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Update: on reflection, I changed wait_for_idle after all. It now reads state and buffered_input_records from the same /stats response. For a paused pipeline, it counts buffered input as processed (input == processed + buffered), and it resets the idle interval when the buffered count changes. A running pipeline uses the old rule. /stats now reports the same state as /status, so a reported pause means that no step takes that input. New unit tests: python/tests/unit/test_wait_for_idle.py. The paused case fails with the old rule.

-- Claude Code, replying at @blp's request.

@mythical-fred-oss

Copy link
Copy Markdown
Contributor

What I ran (debug, static-connectors --test connector_suites)

Suite Result
controller:: 121/121 pass
server:: 38/39 pass. kafka_server fails because it needs a broker
paused_pipeline_does_not_poll_buffered_input ×30 30/30 pass, no flakes

missing (medium): forced HTTP input while paused. Only kafka_server, which needs a broker, tests that a paused single-host pipeline still processes force=true HTTP input. If the is_http_input() exemption broke, normal CI would not catch it. I wrote this test and it passes (3/3 runs):

+    /// A paused pipeline still processes forced HTTP input, and `/status`
+    /// reports `Paused` throughout.
+    #[engine_test]
+    pub fn paused_pipeline_processes_forced_http_input(engine: &dyn PipelineTestEngine) {
+        actix_web::rt::System::new().block_on(async {
+            crate::ensure_default_crypto_provider();
+            let server = start_test_server(
+                engine,
+                &json!({"name": "test", "inputs": {}, "outputs": {}}),
+                Uuid::new_v4(),
+            )
+            .await;
+            start_pipeline(&server).await;
+            send_input(&server, &test_batches(0, 10)).await;
+            pause_pipeline(&server).await;
+            let CompletionTokenResponse { token } =
+                TestHttpSender::send_stream_deserialize_resp::<CompletionTokenResponse>(
+                    server.post("/ingress/test_input1?force=true"),
+                    &test_batches(10, 10),
+                )
+                .await;
+            wait_for_completion(&server, &token).await;
+            assert_eq!(get_stats(&server).await.global_metrics.total_processed_records, 20);
+            wait_for_status(&server, RuntimeStatus::Paused).await;
+        })
+    }

Other uncovered cases (medium):

  • A pipeline that has never run (has_run == false) and is paused still polls all endpoints.
  • A checkpoint or suspend while paused without FT takes the CheckpointBarriers path.

None of these has a test.

Gates: tests, flakiness, docs and the breaking-change changelog entry all pass. The two inline comments cover the /stats vs /status mismatch and the wait_for_idle() hang after a pause.

@blp
blp force-pushed the inputs-quiescent branch from fb82f18 to 365aa7f Compare November 11, 2026 20:12
@blp blp added the multihost Related to multihost or distributed pipelines label Nov 11, 2026
@blp
blp force-pushed the inputs-quiescent branch 2 times, most recently from 42413b8 to 2ac9938 Compare November 11, 2026 22:22
@blp

blp commented Nov 11, 2026

Copy link
Copy Markdown
Member Author

Thanks. Changes for the uncovered cases:

Case Test
Forced HTTP input while paused paused_pipeline_processes_forced_http_input (your test). It fails if paused steps do not poll HTTP endpoints.
Paused before the pipeline first runs (has_run == false) paused_pipeline_polls_until_it_first_runs. It fails without the has_run check. The Delta Lake suspend/resume tests also failed without it.
Checkpoint while paused, without FT No new test. next_step_selection chooses CheckpointBarriers before it looks at the paused state, so this path is the same as before the PR. A test needs a connector that clears its barrier, which the mock connectors cannot do.

The controller::, server:: and transport::http suites pass (161), except kafka_server, which needs a broker.

-- Claude Code, replying at @blp's request.

@blp
blp force-pushed the inputs-quiescent branch from 2ac9938 to bcebd9f Compare November 11, 2026 23:38
@blp

blp commented Nov 11, 2026

Copy link
Copy Markdown
Member Author

Update on the uncovered cases:

Case Status
Checkpoint while paused, without FT New test paused_suspend_polls_barrier_input. A mock connector holds a barrier, then buffers input past it while paused. The suspend must take that input. The test fails if the paused selection takes precedence over CheckpointBarriers.
pause(); wait_for_idle() Fixed in the SDK (see the inline thread).

The full local connector run, with Postgres, Redis, DynamoDB and Redpanda, is still in progress. feldera_adapterlib (73), dbsp_connectors (866) and static_connectors (85) pass.

-- Claude Code, replying at @blp's request.

@blp

blp commented Nov 12, 2026

Copy link
Copy Markdown
Member Author

End-to-end check on a local k3d cluster. A pipeline has a datagen table (200 rows/s) and an HTTP-only table. The script starts it, pauses, waits 5 s, sends 10 rows with input_json(force=True), calls wait_for_idle(), and resumes.

Hosts 0.375.0 (before) This build (cloud e77c31f93, feldera bcebd9fe8)
1 Datagen rows kept arriving while paused (1800 → 2200) No rows while paused. Forced rows processed in 0.2 s. wait_for_idle() returned. Rows resumed after resume().
2 Forced rows not processed within 60 s Same as 1 host: all checks pass, forced rows in 0.2 s.

-- Claude Code, replying at @blp's request.

@blp
blp force-pushed the inputs-quiescent branch 2 times, most recently from 91c9437 to 3a86863 Compare November 13, 2026 15:55
@blp
blp force-pushed the inputs-quiescent branch from 3a86863 to 6df3c62 Compare November 13, 2026 16:29
@blp
blp changed the base branch from main to step-inputs-paused November 13, 2026 16:29
@blp
blp marked this pull request as ready for review November 13, 2026 16:48
@blp
blp force-pushed the step-inputs-paused branch from 4805223 to 3aafb87 Compare November 13, 2026 17:06
@blp
blp force-pushed the inputs-quiescent branch from 6df3c62 to 9300897 Compare November 13, 2026 17:09
| Accepts input from the SDK only when the input is forced | `pipeline.input_json()` raises a `RuntimeError` unless the call passes `force=True`. |
| Does not advance the clock | While the pipeline is paused, the clock connector does not initiate new steps, as it does normally. The value returned by `NOW()` changes in steps initiated by another reason, for example to process buffered or forced input. Such a step *may* advance the clock: either reading the system clock, or, (when `now_http_driven` is configured) to the value supplied by the last `advance_clock()` call executed. |
| Accepts input from the SDK only when the input is forced | `pipeline.input_json()` raises a `RuntimeError` unless the call passes `force=True`. The pipeline processes forced input, and any other input that the HTTP input connectors buffered. |
| Does not advance the clock | The clock connector does not initiate new steps, and the pipeline does not take new clock values from it. `NOW()` keeps its value until `resume()`. |

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pitfall (medium): With now_http_driven, advance_clock() while paused still returns the new now_ms and requests a step, but that step skips the clock endpoint. NOW() doesn't move until resume(). The old row documented the opposite, and the Pipeline.advance_clock() docstring still says the value is emitted "on its next pipeline step" and suggests polling the view. Polling would hang while paused. Please update that docstring and spell out the advance_clock() case here. I confirmed this with a scratch test:

pipeline.pause();
wait(|| status.global_metrics.reported_state() == PipelineState::Paused, 10_000).unwrap();
let steps = status.global_metrics.total_completed_steps();
assert_eq!(advance(&pipeline, Some(60_000)).unwrap(), millis(HTTP_ANCHOR) + 60_000);
wait(|| status.global_metrics.total_completed_steps() > steps, 10_000).unwrap();
assert_eq!(ticks(&pipeline).and_then(|(_, l)| l), Some(millis(HTTP_ANCHOR))); // unchanged
pipeline.start();
wait_for_clock_timestamp(&pipeline, millis(HTTP_ANCHOR) + 60_000);       // applied on resume

It passes. Consider adding it to transport/clock.rs as a regression test for the new contract.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done, in bd09a8304:

  • New regression test advance_http_driven_while_paused in transport/clock.rs, based on your scratch test. It fails if a paused step polls the clock endpoint.
  • The advance_clock() docstring now says that a paused pipeline takes the new value only after resume(), and that polling the view for it must wait until then.
  • The clock row in testing.md now covers advance_clock() with now_http_driven.

-- Claude Code, replying at @blp's request.

@mythical-fred-oss

Copy link
Copy Markdown
Contributor

What I ran

Check Result
connector_suites -- paused (13 tests, including the 4 new ones) ✅ pass
The 4 new tests, run 5 more times ✅ pass every run (~0.35 s)
connector_suites -- controller:: server:: transport::clock transport::http ✅ 172 passed, 1 failed: kafka_server, which needs a broker I couldn't run
uv run --locked pytest tests/unit ✅ 435 passed
pre-commit run --files <changed> ✅ pass
Scratch test: advance_clock while paused (see inline comment) ✅ passes, but the SDK docstring contradicts the new behaviour

Gates: tests, docs and changelog are present. The breaking change is called out. I found no flakiness: the new tests wait on step counters, not sleeps. No new dependencies and no unsafe.

Not covered by tests:

  • missing (medium): HTTP-driven clock advance while paused. See the inline comment.
  • missing (medium): the has_run exception. A pipeline that starts paused, for example by start_paused() or a restart from a checkpoint, polls every connector until it first runs. testing.md lists only the checkpoint exception, so the statement that a paused pipeline "does not process input that its connectors buffer" is not quite true there. Please document this case too.
  • pitfall (medium): wait_for_idle() counts all buffered_input_records as held when paused. Forced HTTP input that is still buffered (input_json(force=True, wait=False)) is about to be processed, yet counts as done, so wait_for_idle() can return early. Worth a unit case or a note.

@blp
blp force-pushed the inputs-quiescent branch from 9300897 to bd09a83 Compare November 13, 2026 18:41
@blp

blp commented Nov 13, 2026

Copy link
Copy Markdown
Member Author

Thanks. Changes in bd09a8304:

Point Change
HTTP-driven clock advance while paused See the inline thread: new test advance_http_driven_while_paused, plus docstring and testing.md updates.
has_run exception testing.md now lists two exceptions in a table: the checkpoint case, and a pipeline that starts paused (start_paused() or a restart from a checkpoint), which polls all connectors until it first runs.
wait_for_idle() with buffered forced HTTP input Fixed in code. For a paused pipeline, input buffered in HTTP input endpoints (.api-ingress-) no longer counts as held, so wait_for_idle() waits for it. New unit case test_paused_pipeline_with_buffered_http_input_is_not_idle, which fails with the old rule.

Python unit tests (436), the clock tests and pre-commit pass.

-- Claude Code, replying at @blp's request.

@blp
blp force-pushed the step-inputs-paused branch 2 times, most recently from 7cdd61a to 573faaf Compare November 13, 2026 18:49
@ryzhyk

ryzhyk commented Nov 13, 2026

Copy link
Copy Markdown
Contributor

Most questions are around what the pipeline can still do in a PAUSED state.

This is the only actual bug I found:

1. High, regression: an API commit on a paused pipeline never finishes while paused, and the circuit thread spins.
- Evidence (repro 1): an API transaction is open, 100 rows wait in a non-HTTP connector, the pipeline pauses, then the transaction commits.
  - PR: 0 rows committed after 5 s, with 27,043 steps meanwhile.
  - Base: committed in 12 ms, after 10 steps.
- Cause: CommitTarget::drained ends the wait for an endpoint only if one of these holds:
  - its queued rows reached the circuit;
  - it has nothing buffered;
  - a later step polled every endpoint (a StepFeed) and took nothing from it.

  A paused step polls only HTTP endpoints and records no StepFeed, so the wait never ends. Meanwhile ControllerInner::transaction_commit_requested keeps StepTrigger::trigger stepping.
- History:
  - 327fa60b1 fixed the same deadlock for the concurrent-bootstrap cutover: during the cutover, the commit goes ahead without the queued input, and transactions.md documents this.
  - edafd74e4 and ef85a2487 fixed spins caused by earlier narrowings of the polled set. StepSelection covers buffered input but not the commit trigger.
- Fix: while paused, poll the endpoints the commit still waits for, which keeps the rule that a commit includes its queued input. Or apply the cutover exemption to the pause as well. Either change removes the spin.

We can still be committing a transaction in the PAUSED state:

3. Medium, not a regression: the views change after pause() returns, contrary to the docs.
- The sentence that fails, in both the changelog and testing.md: "After pause() returns, the views change only because of forced input." testing.md adds a single exception, for checkpoints.
- Other ways the views change after PAUSED:
  - A commit that was running at the pause continues in later steps. In repro 3, the next step ran with PAUSED reported and the transaction still committing, and then all 200 rows committed. The base does the same.
  - A commit requested before the pause starts after it.
  - Non-forced HTTP input buffered before the pause is processed. The table in testing.md says so, which contradicts the sentence.
  - Until the pipeline first runs (has_run), every connector's input is processed.
- History: 86763e0be fixed a bug of the same shape. It had cleared restoring on a step in the middle of a transaction, so /status ran ahead of the snapshot. The fix (replay_finalize) waits for a step at a transaction boundary.
- Fix: report Paused only once no commit is running or requested. Without fix 1, that rule would make pause() wait forever behind an API commit that cannot finish. The alternative is to weaken the sentence.

Not a regression. Claude considers it a bug that we can freeze the pipeline mid-transaction. This doesn't make things worse than today. It's odd though that we don't pause in the middle of an API-triggered transaction, but we do pause in the middle of a connector-driver transaction.

2. High, but it widens an existing hole: a pause strands a connector transaction whose commit is already queued.
- Already true before this PR: a pause stranded a transaction whose end the reader had not read yet. In probe 5, phase 1, the transaction stays open on both versions and the suspend is blocked on both. What the base still did was take a commit queued before or after the pause; the PR does not.
- Evidence:
  - Repro 2 (start, 95 rows and commit queued before the pause): on the PR the transaction stays open, with 10 rows in the circuit, 85 buffered and none committed. The base commits all 95.
  - Repro 4 (suspend in that state): the PR blocks on TemporarySuspendError::TransactionInProgress; the base suspends in 109 ms.
  - Probe 5, phase 2 (commit queued after the pause, as Delta does when the pause lands during a transaction's last batch): the base commits 20 rows and suspends; the PR stays blocked.
- Consequences while the transaction is stranded:
  - no output for the rows already in the circuit;
  - checkpoints and suspend wait until resume, so a non-forced /stop waits too. I did not check whether the manager eventually forces it.
  - forced HTTP input joins the open transaction and does not complete (inferred);
  - wait_for_idle() returns while the transaction is open, because total_processed_records counts uncommitted rows and the PR counts held rows as processed. In repro 2 that is 95 = 10 + 85 (inferred).
- No warning in any of these cases. ControllerInner::paused_transaction_connectors, added by 9a765c711, counts only connectors that the user paused. That gap predates the PR.
- Cause: the Paused branch of CircuitThread::next_step_selection ignores the open transaction. The rule from 20db2c6bb, "don't pause inputs inside a transaction", survives in two places: the TransactionState::None guard on the CheckpointBarriers branch, and the comment above the call in CircuitThread::input_step.
- Fix: to get back to the base's behavior, poll the connectors still Started in TransactionInitiators::initiated_by_connectors while paused, through StepSelection. Closing the older hole as well would mean that BackpressureThread::backpressure_thread keeps such a connector running until it queues its commit. A pause would then last as long as the transaction, and a Delta snapshot transaction can run for hours. Either way, extend the warning to a pipeline pause.

@ryzhyk ryzhyk left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's unclear how /pause should interact with transactions. We have two options: (1) preserve the current behavior by addressing the first issue I reported. This is the minimal fix that will make the PR acceptable. (2) figure out a more reasonable/consistent behavior and implement that. I'm ok either way.

Base automatically changed from step-inputs-paused to main November 13, 2026 23:42
@blp
blp force-pushed the inputs-quiescent branch from bd09a83 to fbe69e1 Compare November 13, 2026 23:51
@blp

blp commented Nov 14, 2026

Copy link
Copy Markdown
Member Author

Thanks. This takes option 1, with the first fix from issue 1. Changes in b29394dbd:

Issue Change
1. API commit never finishes while paused While an API commit waits for queued input (commit_target is set), next_step_selection polls all input endpoints, even if the pipeline is paused. The step records a StepFeed, so CommitTarget::drained can finish. When the commit starts, the pause applies again. New test paused_api_transaction_commit_includes_queued_records. Without the fix, it fails with 0 of 100 records committed.
2. Pause strands a connector transaction with a queued commit Not changed.
3. Docs say that views change only because of forced input The sentence in testing.md and the changelog now allows HTTP input and the listed cases. The exceptions table in testing.md has a new row for a transaction commit. transactions.md says that a paused pipeline still takes the queued input for a commit.

connector_suites passes for controller::, paused and api_transaction (131 tests), and for server::, transport::http and transport::clock (48 tests, without kafka_server, which needs a broker).

-- Claude Code, replying at @blp's request.

@blp
blp requested a review from ryzhyk November 16, 2026 18:13

@ryzhyk ryzhyk left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the fixes!

Claude recommends adjusting wording in docs in several places to match the actual behavior:

Doc and current wording: testing.md, clock row: "The clock connector does not initiate new steps, and the pipeline does not take new clock values from it. NOW() keeps its value until resume(). With now_http_driven, advance_clock()
  returns the new value, but the pipeline takes it only after resume()."
Verdict: Broken
Evidence: Probe 7: NOW() moved to the advanced value while /status said Paused.
Proposed wording: "The clock connector does not initiate new steps, and NOW() keeps its value until resume(), with one exception: from a commit_transaction() request until the commit starts, the pipeline takes a new clock value
  (see below). With now_http_driven, advance_clock() returns the new value at once. The pipeline takes it after resume() or during that exception, whichever comes first."
────────────────────────────────────────
Doc and current wording: SDK Pipeline.advance_clock() docstring: "A paused pipeline does not take the new value until it resumes, so while the pipeline is paused, the views keep the previous NOW(). Do not poll the view for the new
  value until after :meth:.resume."
Verdict: Broken
Evidence: Probe 7
Proposed wording: "A paused pipeline takes the new value when it resumes, or earlier if it commits a transaction: from a :meth:.commit_transaction request until the commit starts, it polls all its connectors, the clock included.
  Otherwise, while the pipeline is paused, the views keep the previous NOW(), so do not poll the view for the new value until after :meth:.resume."
────────────────────────────────────────
Doc and current wording: testing.md, "Transaction commit" row: "A transaction that commits while the pipeline is paused includes the input that the connectors buffered before the commit request, so the pipeline processes that input
  to finish the commit. A commit that was in progress at the pause also finishes."
Verdict: Understated
Evidence: Probe 8: rows queued after the request went in while paused, and the commit was still blocked on a joined connector. Repro 2: a transaction that a connector started stayed open, 0 of 95 rows committed.
Proposed wording: "From a commit_transaction() request until the commit starts, a paused pipeline polls all its connectors, the clock included, as a running pipeline does. The transaction thus includes the input that the connectors
  buffered before the request, and any input that arrives before the commit starts. If a connector in the transaction has not committed its part, this lasts until it does, which can be after resume(). A commit that was in progress
  at the pause also finishes. A transaction that only connectors started does not commit while the pipeline is paused, unless its commit started before the pause. Until it commits, the pipeline takes no checkpoints, and a suspend
  waits."
────────────────────────────────────────
Doc and current wording: testing.md, paragraph above the table: "A connector can continue to buffer input for a short time after the pause, but the pipeline does not process that input until resume()."
Verdict: Understated (same cause as the row above)
Evidence: Probe 8
Proposed wording: "A connector can continue to buffer input for a short time after the pause. Except in the cases in the table below, the pipeline does not process that input until resume()."
────────────────────────────────────────
Doc and current wording: testing.md, below the table: "A multihost pipeline behaves the same way."
Verdict: Understated (inferred)
Evidence: ControllerInner::start_commit_transaction_from_api sets commit_target only for a single-host pipeline: "the coordinator commits with steps that feed no input". I did not check the coordinator.
Proposed wording: "A multihost pipeline behaves the same way, except that a transaction commit does not take the input that the connectors buffered."
────────────────────────────────────────
Doc and current wording: lifecycle.md, Paused row: "The input connectors are paused, and the pipeline does not process the input that they buffered. Only forced input on HTTP input connectors goes in."
Verdict: Stale
Evidence: Lists no exceptions. Non-forced HTTP input buffered before the pause also goes in.
Proposed wording: "The input connectors are paused, and the pipeline does not process the input that they buffered. The exception is input in the HTTP input connectors, which accept new input while paused only with force=true. For
  other exceptions, such as a transaction commit or a checkpoint, see Pausing a pipeline (/tutorials/testing#pausing-a-pipeline)."
────────────────────────────────────────
Doc and current wording: SDK Pipeline.pause() docstring: "A paused pipeline does not process the input that its connectors buffered, except forced input to HTTP input connectors. When this function waits, it returns after the step
  that was in progress at the time of the pause is complete."
Verdict: Stale
Evidence: Lists no exception besides HTTP. Repro 3: a commit running at the pause changes the views after pause() returns.
Proposed wording: "A paused pipeline does not process the input that its connectors buffered. The exceptions are input in HTTP input connectors, such as forced input from :meth:.input_json, and the cases listed under "Pausing a
  pipeline" in the testing guide, such as a transaction commit. When this function waits, it returns after the step that was in progress at the time of the pause is complete. A transaction commit that was in progress at the pause
  still finishes after that, which changes the views."
────────────────────────────────────────
Doc and current wording: PR description: "A reported pause thus means that no step takes connector input." Its exceptions list has has_run, checkpoint and coordinator.
Verdict: Stale
Evidence: The exceptions list lacks the commit exception from b29394dbd.
Proposed wording: Sentence: "A reported pause thus means that no step takes input from connectors other than HTTP input endpoints, except in the cases below." New bullet: "From an API commit_transaction request until the commit
  starts, steps poll all endpoints, so that the commit includes the input queued at the request. This lasts until every connector in the transaction has committed its part. A commit in progress at the pause also finishes."

@blp

blp commented Nov 16, 2026

Copy link
Copy Markdown
Member Author

Thanks. 26a2fa941 applies the proposed wording for:

Place Change
testing.md, clock row NOW() can change from a commit_transaction() request until the commit starts
testing.md, paragraph above the table Adds "except in the cases in the table below"
testing.md, "Transaction commit" row Input that arrives until the commit starts; a connector that has not committed its part; a transaction that only connectors started
testing.md, multihost sentence A multihost commit does not take buffered input
lifecycle.md, Paused row HTTP input, and a link to the other exceptions
SDK pause() and advance_clock() docstrings The commit exception

The PR description now has the commit exception, and its pause sentence allows HTTP input and the listed cases.

-- Claude Code, replying at @blp's request.

blp added 4 commits November 16, 2026 11:52
Before, a paused single-host pipeline continued to poll its input
connectors, so input that they buffered before or shortly after the pause
still went into the circuit.  No API could confirm that input had stopped,
which made tests nondeterministic.

Now, without a coordinator, a paused pipeline steps with
`StepInputs::Paused`, which polls only HTTP input endpoints (they accept
input while paused only with `force=true`), and during a checkpoint, with
`StepInputs::CheckpointBarriers` as before.  The circuit thread records the
state that it applied, and the pipeline reports `Paused` only after it
applies the pause, so no step in progress can take more input.  A paused
pipeline that has never run still polls every endpoint once, so that each
connector records its starting position for checkpoints.  With
feldera/cloud#2209, a multihost pipeline behaves the same way.

For a paused pipeline, the Python SDK's `wait_for_idle()` counts the input
that connectors hold in buffers as if it were processed, because the
pipeline does not process it until it resumes.  The exception is input in
HTTP input connectors, which a paused pipeline still processes.

With `now_http_driven`, `advance_clock()` while paused returns the new
`NOW()`, but the pipeline takes it only after it resumes.

Fixes #7568.

Signed-off-by: Ben Pfaff <blp@feldera.com>
An API commit includes the input that connectors queued before the commit
request, so it waits until a step polls every input endpoint.  A paused
pipeline polled only HTTP input endpoints, so the commit never finished,
and the circuit thread kept stepping to advance it.

Now, while an API commit waits for queued input, a paused pipeline polls
all its input endpoints.  When the commit starts, the pause applies again.
The docs now list a transaction commit as a case in which a paused
pipeline processes buffered input.

Signed-off-by: Ben Pfaff <blp@feldera.com>
From a `commit_transaction()` request until the commit starts, a paused
pipeline polls all its connectors, the clock included.  The docs for the
clock, `advance_clock()`, `pause()` and the Paused state now say so, and
also that a transaction that only connectors started does not commit
while the pipeline is paused.  A multihost commit does not take buffered
input.

Signed-off-by: Ben Pfaff <blp@feldera.com>
v0.383.0 was released before this change merged.

Signed-off-by: Ben Pfaff <blp@feldera.com>
@blp
blp force-pushed the inputs-quiescent branch from 26a2fa9 to 0db9bfe Compare November 16, 2026 19:56
@blp
blp enabled auto-merge November 16, 2026 19:57
@blp
blp disabled auto-merge November 16, 2026 20:10
@blp
blp added this pull request to the merge queue Nov 16, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Nov 16, 2026
@blp
blp added this pull request to the merge queue Nov 16, 2026
Merged via the queue into main with commit 2cc28ec Nov 17, 2026
1 check passed
@blp
blp deleted the inputs-quiescent branch November 17, 2026 02:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working connectors Issues related to the adapters/connectors crate documentation Improvements or additions to documentation multihost Related to multihost or distributed pipelines User-facing For PRs that lead to Feldera-user visible changes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[manager] There is no way to check that all input connectors are paused

3 participants