← 输入框|github.com|刷新
Skip to content

[adapters] Report a host's committed input records in its step status - #7607

Merged
blp merged 2 commits into
mainfrom
multihost-committed-records
Oct 29, 2026
Merged

blp merged 2 commits into
mainfrom
multihost-committed-records

Conversation

@blp

@blp blp commented Oct 23, 2026

Copy link
Copy Markdown
Member

In a multihost pipeline, each host counts only its own input. An output connector's total_processed_input_records, and the total_completed_records that a host derives from it, therefore count only the input of the connector's own host. If the input arrives at a different host from the connector, the connector's count does not advance at all, so test_bootstrapping.py (silent and concurrent bootstrap) times out with 2 hosts.

Now each host reports, in /coordination/step/status, the number of input records that it has committed as of the step (StepStatus::total_committed_records). The coordinator adds up the hosts' counts (feldera/cloud PR to follow). The field is optional, so an older coordinator ignores it.

Change Location
New optional field total_committed_records crates/feldera-types/src/coordination.rs
The first status carries the count restored from the checkpoint; after each step, the host records the step's processed records before it reports Idle, so the report carries this step's count crates/adapters/src/controller.rs
Doc comments: these metrics now count every host in a multihost pipeline (plus openapi.json, types.gen.ts) crates/feldera-types/src/adapter_stats.rs, crates/adapters/src/controller/stats.rs
Test step_status_reports_committed_records crates/adapters/src/controller/test.rs

Tested: the new test passes, and fails (Some(0) instead of Some(5)) if the host reports the count from before the step commits. The adapters controller:: and server:: tests pass, except server::test_with_kafka::test_server, which needs a Kafka broker. Not yet tested end to end with the coordinator change on a multihost cluster.

@blp blp self-assigned this Oct 23, 2026
@blp blp added bug Something isn't working ft Fault tolerant, distributed, and scale-out implementation connectors Issues related to the adapters/connectors crate rust Pull requests that update Rust code multihost Related to multihost or distributed pipelines labels Oct 23, 2026
/// Number of input records that this host has committed. The input of a
/// step commits when the step ends, except that the input of a transaction
/// commits only when the transaction commits, so the input of an open
/// transaction does not count yet. Reported while `Idle`, when it is the

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't understand the last phrase.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, that was unclear. Reworded in 649c5af:

If action is [StepAction::Idle], the number of input records that this host committed in the steps before step; otherwise, None.

The rule about transactions is now its own paragraph.

/// count as of `step`.
///
/// The coordinator adds up the hosts' counts for a step to get the number
/// for the whole pipeline. `None` if the host does not report it.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the host? The previous phrase uses a plural.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, it was ambiguous after the plural. Fixed in 649c5af:

The coordinator adds up the counts that all of the hosts report for the same step to get the count for the whole pipeline. A host that runs an older version does not report a count, so this is always None in the status that it sends.

/// This is meaningful in a multihost pipeline because steps are
/// synchronized across all of the hosts.
/// In a multihost pipeline, the hosts run the same steps together, so this
/// count means the same as with one host.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not thrilled about this phrase; it seems to be a circular definition.
I would rather say what the number really represents.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fair point. In 649c5af, this now says what a step number is:

In a multihost pipeline, all of the hosts run the same steps together, so a step number refers to the same step on every host.

total_processed_input_records had the same circular phrase, so it now says what N counts: the input records that all of the hosts received (and only the endpoint's own host's input if a host runs an older version).

///
/// In a multihost pipeline, this counts the input records of all of the
/// hosts, as an output endpoint's `total_processed_input_records` does, and
/// has the same limit with a host that does not report its count.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this can be interpreted incorrectly

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed. In 649c5af, it now says exactly what happens with an older host:

In a multihost pipeline, this counts the input records that all of the hosts received. If a host runs an older version that does not report its count to the coordinator, this is instead the sum of the counts that each host reports for its own input, which can include input whose output an output connector on a different host has not yet processed.

/// processed on the same host as the output endpoint, which is not usually
/// meaningful.
/// In a multihost pipeline, each host counts only its own input, but the
/// coordinator replaces this with the input records of all of the hosts as

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pitfall (medium): This becomes public API docs (openapi.json, types.gen.ts) as soon as this PR merges, but nothing in this repo does the replacement yet; the coordinator side is a separate change. Until that ships, multihost users will read that the count covers every host when it still covers only one. Could this doc change land with, or after, the coordinator change? Or say "a coordinator that supports it replaces…".

@mythical-fred-oss

Copy link
Copy Markdown
Contributor

What I ran

Check Result
cargo test -p dbsp_adapters --lib step_status_reports_committed_records ✅ (15/15 repeated runs)
pre-commit run on the changed Rust files ✅
QA tests I added locally (diff below) ✅ (15/15 repeated runs)

The reorder in controller.rs looks correct to me. set_committed now runs before the Idle report, and transaction_status still comes from the state at the start of the step. The new field is Option with serde(default, skip_serializing_if), so older peers keep working in both directions.

missing (medium): tests. The PR's test covers only the plain, non-transactional path. These cases weren't covered:

  1. Records inside an open transaction must not count until the transaction commits. This is the subtle part of the doc comment. Added below; it passes.
  2. Wire compatibility: an older host omits the field, and None isn't serialized. Added below; it passes.
  3. After a restart from a checkpoint, the first Idle should report the restored count, which is non-zero. Not covered, and I didn't write it.
QA test diff
diff --git a/crates/adapters/src/controller/test.rs b/crates/adapters/src/controller/test.rs
index a5dc24e..af59d1d 100644
--- a/crates/adapters/src/controller/test.rs
+++ b/crates/adapters/src/controller/test.rs
@@ -10249,3 +10249,75 @@ fn paused_connector_holds_transaction() {
     assert!(paused().is_empty());
     controller.stop().unwrap();
 }
+
+/// Records ingested inside an open transaction are not committed, so the step
+/// status must not count them until the transaction commits.
+#[test]
+fn step_status_excludes_open_transaction() {
+    init_test_logger();
+
+    let config: PipelineConfig = serde_json::from_value(json!({
+        "name": "test",
+        "workers": 4,
+        "min_batch_size_records": 0,
+        "max_buffering_delay_usecs": 0,
+        "clock_resolution_usecs": null,
+    }))
+    .unwrap();
+    let controller = Controller::with_test_config(
+        |circuit_config| Ok(test_circuit_with_aggregate(circuit_config, 16)),
+        &config,
+        Box::new(|e, _| panic!("error: {e}")),
+    )
+    .unwrap();
+    let step_watcher = controller.step_watcher();
+    controller.start();
+
+    let feeder = TransactionalInputEndpoint::new();
+    let endpoint_config: InputEndpointConfig = serde_json::from_value(json!({
+        "stream": "test_input1",
+        "transport": { "name": "file_input", "config": { "path": "unused" } },
+        "format": { "name": "csv" }
+    }))
+    .unwrap();
+    controller
+        .add_input_endpoint("feeder", endpoint_config, Box::new(feeder.clone()), None)
+        .unwrap();
+
+    const TIMEOUT_MS: u128 = 20_000;
+    const RECORDS: usize = 10;
+    controller.start_transaction().unwrap();
+    feeder.push(
+        &(0..RECORDS)
+            .map(|id| format!("{id},true,{id},record-{id}\n"))
+            .collect::<String>(),
+    );
+    wait(
+        || controller.status().num_total_circuit_input_records() >= RECORDS as u64,
+        TIMEOUT_MS,
+    )
+    .unwrap();
+    let ingested_step = step_watcher.borrow().step;
+    wait(
+        || {
+            let s = *step_watcher.borrow();
+            s.action == StepAction::Idle && s.step >= ingested_step
+        },
+        TIMEOUT_MS,
+    )
+    .unwrap();
+    let status = *step_watcher.borrow();
+    assert_eq!(status.transaction_status, Some(true), "{status:?}");
+    assert_eq!(status.total_committed_records, Some(0), "{status:?}");
+
+    controller.start_commit_transaction().unwrap();
+    wait(
+        || {
+            let s = *step_watcher.borrow();
+            s.action == StepAction::Idle && s.total_committed_records == Some(RECORDS as u64)
+        },
+        TIMEOUT_MS,
+    )
+    .unwrap_or_else(|_| panic!("last step status: {:?}", *step_watcher.borrow()));
+    controller.stop().unwrap();
+}
diff --git a/crates/feldera-types/src/coordination.rs b/crates/feldera-types/src/coordination.rs
index dc605e2..63677d2 100644
--- a/crates/feldera-types/src/coordination.rs
+++ b/crates/feldera-types/src/coordination.rs
@@ -780,3 +780,22 @@ mod tests {
         assert!(CoordinationCapabilities::current().distributed_inputs);
     }
 }
+
+#[cfg(test)]
+mod step_status_serde_tests {
+    use super::*;
+
+    #[test]
+    fn total_committed_records_is_optional_on_the_wire() {
+        // An older host omits the field.
+        let old: StepStatus = serde_json::from_str(r#"{"step":3,"action":"idle"}"#).unwrap();
+        assert_eq!(old.total_committed_records, None);
+        // `None` is not serialized, so an older coordinator sees the old shape.
+        let json = serde_json::to_value(StepStatus::new(3, StepAction::Idle, None)).unwrap();
+        assert!(json.get("total_committed_records").is_none());
+        // A new host's count round-trips.
+        let new = StepStatus::new(3, StepAction::Idle, None).with_total_committed_records(7);
+        let back: StepStatus = serde_json::from_value(serde_json::to_value(new).unwrap()).unwrap();
+        assert_eq!(back.total_committed_records, Some(7));
+    }
+}

@ryzhyk

ryzhyk commented Oct 26, 2026

Copy link
Copy Markdown
Contributor

The fix looks simpler than I expected it to be, probably because we already had a mechanism to delay transaction commit until all connectors agree to commit.

I asked claude to look for races that may cause incorrect transaction boundaries or deadlocks. Naturally it found a couple, the second one being more serious.

Proposed fixes here: https://github.com/feldera/feldera/tree/early-commit-keeps-transaction-fixes (WARNING: I haven't looked)

  1. A commit during a step can still drop the transaction's records (the fix is incomplete)

CommitTarget::drained (controller.rs:10082) treats an endpoint as done if this step fed it 0 records, or didn't poll it. But the step may have polled that endpoint before the commit request arrived:

  1. The circuit thread, in CircuitThread::input_step, polls feeder. Its queue is empty, so it feeds 0 records. The thread then waits for a slower endpoint.
  2. The client sends 100 records to feeder and calls /commit_transaction. The commit target becomes {feeder: 100}.
  3. The circuit thread reaches advance_transaction_state with the old feed {feeder: 0}. The target counts as drained, and the transaction commits without the 100 records. Those records then go into later automatic transactions, the very split the PR fixes.

Three variants, each with its own test (a test endpoint holds the step between the two polls):

  • the transaction has already started;
  • no step has started it yet;
  • the records go to an endpoint created after the step began, as the first /ingress for a table does. An endpoint the step never polled also counts as done.

All three fail on the PR and on its parent with the transaction committed with 0 of its 100 records. The race window grows with any slow endpoint, such as an /ingress endpoint whose lock a large concurrent upload holds, or a Kafka or Delta flush. Antithesis will find it.

Suggested fix (tested): number each step's poll under the transaction_info lock and store the current number in CommitTarget at the commit request. Then accept the "fed nothing" exit only from a later poll. This costs at most one extra step.

  1. The PR causes a hang during a concurrent bootstrap
  • In the cutover window (phase Synchronizing: AwaitingSync through Finalizing), input_step returns early without reading input (controller.rs:4514), and the backpressure thread pauses every endpoint.
  • AwaitingSync and Finalizing wait for TransactionState::None (controller.rs:3655, controller.rs:3695).
  • Controller::start_transaction and start_commit_transaction reject only a stop-the-world bootstrap or a restore, so API transactions run during a concurrent bootstrap.

So if the API commits while any endpoint has queued records, the target can never drain. Kafka nearly always has records queued, and so does /ingress input that arrived just before the pause. The transaction never leaves Started, the cutover waits forever, and the trigger forces an empty step on every loop.

I wrote an end-to-end test with the existing concurrent-bootstrap harness, using /ingress?force=true to queue records during the pause:

  • PR: after 20 s, the transaction is TransactionInProgress and the bootstrap is Synchronizing, after 115092 steps
  • Parent: passes in 0.72 s.

The PR's claim that "the wait always ends" is false whenever steps poll no endpoints at all.

Fix options:

  • (a) Tested: stop waiting for the target when the step skipped reading input because of the cutover pause. Queued records then go to the next transaction, as before the PR, but only in this window.
  • (b) Keeps the guarantee: read input during AwaitingSync and Finalizing while a commit target is pending. Whoever owns concurrent bootstrapping would need to confirm this is safe for the recorders.

With both prototype fixes applied, the controller:: tests pass (173 passed, 0 failed), as do the five concurrent-bootstrap tests. The new race tests passed 5 of 5 runs.

@ryzhyk ryzhyk left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think at least the second of the two issues claude found needs to be addressed

@blp

blp commented Oct 26, 2026

Copy link
Copy Markdown
Member Author

Every change and every bug fix makes the controller more baffling.

@blp

blp commented Oct 26, 2026

Copy link
Copy Markdown
Member Author

@ryzhyk I've studied the fixes by hand and they make sense to me. I'll ask claude to take a look too.

@blp

blp commented Oct 26, 2026

Copy link
Copy Markdown
Member Author

@ryzhyk's review here, and my responses, were meant for #7583.

@ryzhyk

ryzhyk commented Oct 26, 2026

Copy link
Copy Markdown
Contributor

@ryzhyk's review here, and my responses, were meant for #7583.

oops, sorry about that

blp added 2 commits October 28, 2026 10:37
In a multihost pipeline, each host counts only its own input.  An output
connector's `total_processed_input_records`, and the
`total_completed_records` that a host derives from it, therefore count
only the input of the connector's own host, although the connector
writes the output of every host's input.  If the input arrives at a
different host from the connector, the connector's count does not
advance at all.

Now each host reports, in `/coordination/step/status`, the number of
input records it has committed as of the step, so that the coordinator
can add up the hosts' counts.  The field is optional, so a coordinator
that does not know it ignores it.

To report the count for the step that just ran, the host now records
the step's processed records before it reports `Idle`, not after.

The doc comments on these metrics said that, in a multihost pipeline,
`total_processed_input_records` is not usually meaningful.  Now they say
that the coordinator reports the count for all of the hosts, and which
hosts still get the old behavior.
Review comments said that some of these comments were unclear:

- `StepStatus::total_committed_records` said "Reported while `Idle`,
  when it is the count as of `step`", and "the host" after a sentence
  about all of the hosts.  Now it says that the count is set only when
  `action` is `Idle`, which input it counts, and which host does not
  report one.

- `total_processed_input_records` and `total_processed_steps` said that
  the count "means the same as with one host", which defines the count
  in terms of itself.  Now they say what the count is.

- `total_completed_records` said that it "has the same limit" with an
  older host.  Now it says what it counts in that case.
@blp
blp force-pushed the multihost-committed-records branch from 649c5af to 7accafb Compare October 28, 2026 22:32
@blp
blp enabled auto-merge October 28, 2026 22:33
@blp
blp disabled auto-merge October 28, 2026 23:32
@blp
blp added this pull request to the merge queue Oct 28, 2026
Merged via the queue into main with commit 75265b5 Oct 29, 2026
1 check passed
@blp
blp deleted the multihost-committed-records branch October 29, 2026 08:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working connectors Issues related to the adapters/connectors crate ft Fault tolerant, distributed, and scale-out implementation multihost Related to multihost or distributed pipelines rust Pull requests that update Rust code

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants