Repository navigation
[adapters] Report a host's committed input records in its step status - #7607
Conversation
| /// Number of input records that this host has committed. The input of a | ||
| /// step commits when the step ends, except that the input of a transaction | ||
| /// commits only when the transaction commits, so the input of an open | ||
| /// transaction does not count yet. Reported while `Idle`, when it is the |
There was a problem hiding this comment.
I don't understand the last phrase.
There was a problem hiding this comment.
Agreed, that was unclear. Reworded in 649c5af:
If
actionis [StepAction::Idle], the number of input records that this host committed in the steps beforestep; otherwise,None.
The rule about transactions is now its own paragraph.
| /// count as of `step`. | ||
| /// | ||
| /// The coordinator adds up the hosts' counts for a step to get the number | ||
| /// for the whole pipeline. `None` if the host does not report it. |
There was a problem hiding this comment.
the host? The previous phrase uses a plural.
There was a problem hiding this comment.
Yes, it was ambiguous after the plural. Fixed in 649c5af:
The coordinator adds up the counts that all of the hosts report for the same
stepto get the count for the whole pipeline. A host that runs an older version does not report a count, so this is alwaysNonein the status that it sends.
| /// This is meaningful in a multihost pipeline because steps are | ||
| /// synchronized across all of the hosts. | ||
| /// In a multihost pipeline, the hosts run the same steps together, so this | ||
| /// count means the same as with one host. |
There was a problem hiding this comment.
I am not thrilled about this phrase; it seems to be a circular definition.
I would rather say what the number really represents.
There was a problem hiding this comment.
Fair point. In 649c5af, this now says what a step number is:
In a multihost pipeline, all of the hosts run the same steps together, so a step number refers to the same step on every host.
total_processed_input_records had the same circular phrase, so it now says what N counts: the input records that all of the hosts received (and only the endpoint's own host's input if a host runs an older version).
| /// | ||
| /// In a multihost pipeline, this counts the input records of all of the | ||
| /// hosts, as an output endpoint's `total_processed_input_records` does, and | ||
| /// has the same limit with a host that does not report its count. |
There was a problem hiding this comment.
I think this can be interpreted incorrectly
There was a problem hiding this comment.
Agreed. In 649c5af, it now says exactly what happens with an older host:
In a multihost pipeline, this counts the input records that all of the hosts received. If a host runs an older version that does not report its count to the coordinator, this is instead the sum of the counts that each host reports for its own input, which can include input whose output an output connector on a different host has not yet processed.
| /// processed on the same host as the output endpoint, which is not usually | ||
| /// meaningful. | ||
| /// In a multihost pipeline, each host counts only its own input, but the | ||
| /// coordinator replaces this with the input records of all of the hosts as |
There was a problem hiding this comment.
pitfall (medium): This becomes public API docs (openapi.json, types.gen.ts) as soon as this PR merges, but nothing in this repo does the replacement yet; the coordinator side is a separate change. Until that ships, multihost users will read that the count covers every host when it still covers only one. Could this doc change land with, or after, the coordinator change? Or say "a coordinator that supports it replaces…".
|
What I ran
The reorder in missing (medium): tests. The PR's test covers only the plain, non-transactional path. These cases weren't covered:
QA test diffdiff --git a/crates/adapters/src/controller/test.rs b/crates/adapters/src/controller/test.rs
index a5dc24e..af59d1d 100644
--- a/crates/adapters/src/controller/test.rs
+++ b/crates/adapters/src/controller/test.rs
@@ -10249,3 +10249,75 @@ fn paused_connector_holds_transaction() {
assert!(paused().is_empty());
controller.stop().unwrap();
}
+
+/// Records ingested inside an open transaction are not committed, so the step
+/// status must not count them until the transaction commits.
+#[test]
+fn step_status_excludes_open_transaction() {
+ init_test_logger();
+
+ let config: PipelineConfig = serde_json::from_value(json!({
+ "name": "test",
+ "workers": 4,
+ "min_batch_size_records": 0,
+ "max_buffering_delay_usecs": 0,
+ "clock_resolution_usecs": null,
+ }))
+ .unwrap();
+ let controller = Controller::with_test_config(
+ |circuit_config| Ok(test_circuit_with_aggregate(circuit_config, 16)),
+ &config,
+ Box::new(|e, _| panic!("error: {e}")),
+ )
+ .unwrap();
+ let step_watcher = controller.step_watcher();
+ controller.start();
+
+ let feeder = TransactionalInputEndpoint::new();
+ let endpoint_config: InputEndpointConfig = serde_json::from_value(json!({
+ "stream": "test_input1",
+ "transport": { "name": "file_input", "config": { "path": "unused" } },
+ "format": { "name": "csv" }
+ }))
+ .unwrap();
+ controller
+ .add_input_endpoint("feeder", endpoint_config, Box::new(feeder.clone()), None)
+ .unwrap();
+
+ const TIMEOUT_MS: u128 = 20_000;
+ const RECORDS: usize = 10;
+ controller.start_transaction().unwrap();
+ feeder.push(
+ &(0..RECORDS)
+ .map(|id| format!("{id},true,{id},record-{id}\n"))
+ .collect::<String>(),
+ );
+ wait(
+ || controller.status().num_total_circuit_input_records() >= RECORDS as u64,
+ TIMEOUT_MS,
+ )
+ .unwrap();
+ let ingested_step = step_watcher.borrow().step;
+ wait(
+ || {
+ let s = *step_watcher.borrow();
+ s.action == StepAction::Idle && s.step >= ingested_step
+ },
+ TIMEOUT_MS,
+ )
+ .unwrap();
+ let status = *step_watcher.borrow();
+ assert_eq!(status.transaction_status, Some(true), "{status:?}");
+ assert_eq!(status.total_committed_records, Some(0), "{status:?}");
+
+ controller.start_commit_transaction().unwrap();
+ wait(
+ || {
+ let s = *step_watcher.borrow();
+ s.action == StepAction::Idle && s.total_committed_records == Some(RECORDS as u64)
+ },
+ TIMEOUT_MS,
+ )
+ .unwrap_or_else(|_| panic!("last step status: {:?}", *step_watcher.borrow()));
+ controller.stop().unwrap();
+}
diff --git a/crates/feldera-types/src/coordination.rs b/crates/feldera-types/src/coordination.rs
index dc605e2..63677d2 100644
--- a/crates/feldera-types/src/coordination.rs
+++ b/crates/feldera-types/src/coordination.rs
@@ -780,3 +780,22 @@ mod tests {
assert!(CoordinationCapabilities::current().distributed_inputs);
}
}
+
+#[cfg(test)]
+mod step_status_serde_tests {
+ use super::*;
+
+ #[test]
+ fn total_committed_records_is_optional_on_the_wire() {
+ // An older host omits the field.
+ let old: StepStatus = serde_json::from_str(r#"{"step":3,"action":"idle"}"#).unwrap();
+ assert_eq!(old.total_committed_records, None);
+ // `None` is not serialized, so an older coordinator sees the old shape.
+ let json = serde_json::to_value(StepStatus::new(3, StepAction::Idle, None)).unwrap();
+ assert!(json.get("total_committed_records").is_none());
+ // A new host's count round-trips.
+ let new = StepStatus::new(3, StepAction::Idle, None).with_total_committed_records(7);
+ let back: StepStatus = serde_json::from_value(serde_json::to_value(new).unwrap()).unwrap();
+ assert_eq!(back.total_committed_records, Some(7));
+ }
+} |
|
The fix looks simpler than I expected it to be, probably because we already had a mechanism to delay transaction commit until all connectors agree to commit. I asked claude to look for races that may cause incorrect transaction boundaries or deadlocks. Naturally it found a couple, the second one being more serious. Proposed fixes here: https://github.com/feldera/feldera/tree/early-commit-keeps-transaction-fixes (WARNING: I haven't looked)
CommitTarget::drained (controller.rs:10082) treats an endpoint as done if this step fed it 0 records, or didn't poll it. But the step may have polled that endpoint before the commit request arrived:
Three variants, each with its own test (a test endpoint holds the step between the two polls):
All three fail on the PR and on its parent with the transaction committed with 0 of its 100 records. The race window grows with any slow endpoint, such as an /ingress endpoint whose lock a large concurrent upload holds, or a Kafka or Delta flush. Antithesis will find it. Suggested fix (tested): number each step's poll under the transaction_info lock and store the current number in CommitTarget at the commit request. Then accept the "fed nothing" exit only from a later poll. This costs at most one extra step.
So if the API commits while any endpoint has queued records, the target can never drain. Kafka nearly always has records queued, and so does /ingress input that arrived just before the pause. The transaction never leaves Started, the cutover waits forever, and the trigger forces an empty step on every loop. I wrote an end-to-end test with the existing concurrent-bootstrap harness, using /ingress?force=true to queue records during the pause:
The PR's claim that "the wait always ends" is false whenever steps poll no endpoints at all. Fix options:
With both prototype fixes applied, the controller:: tests pass (173 passed, 0 failed), as do the five concurrent-bootstrap tests. The new race tests passed 5 of 5 runs. |
ryzhyk
left a comment
There was a problem hiding this comment.
I think at least the second of the two issues claude found needs to be addressed
|
Every change and every bug fix makes the controller more baffling. |
|
@ryzhyk I've studied the fixes by hand and they make sense to me. I'll ask claude to take a look too. |
In a multihost pipeline, each host counts only its own input. An output connector's `total_processed_input_records`, and the `total_completed_records` that a host derives from it, therefore count only the input of the connector's own host, although the connector writes the output of every host's input. If the input arrives at a different host from the connector, the connector's count does not advance at all. Now each host reports, in `/coordination/step/status`, the number of input records it has committed as of the step, so that the coordinator can add up the hosts' counts. The field is optional, so a coordinator that does not know it ignores it. To report the count for the step that just ran, the host now records the step's processed records before it reports `Idle`, not after. The doc comments on these metrics said that, in a multihost pipeline, `total_processed_input_records` is not usually meaningful. Now they say that the coordinator reports the count for all of the hosts, and which hosts still get the old behavior.
Review comments said that some of these comments were unclear: - `StepStatus::total_committed_records` said "Reported while `Idle`, when it is the count as of `step`", and "the host" after a sentence about all of the hosts. Now it says that the count is set only when `action` is `Idle`, which input it counts, and which host does not report one. - `total_processed_input_records` and `total_processed_steps` said that the count "means the same as with one host", which defines the count in terms of itself. Now they say what the count is. - `total_completed_records` said that it "has the same limit" with an older host. Now it says what it counts in that case.
649c5af to
7accafb
Compare
In a multihost pipeline, each host counts only its own input. An output connector's
total_processed_input_records, and thetotal_completed_recordsthat a host derives from it, therefore count only the input of the connector's own host. If the input arrives at a different host from the connector, the connector's count does not advance at all, sotest_bootstrapping.py(silent and concurrent bootstrap) times out with 2 hosts.Now each host reports, in
/coordination/step/status, the number of input records that it has committed as of the step (StepStatus::total_committed_records). The coordinator adds up the hosts' counts (feldera/cloud PR to follow). The field is optional, so an older coordinator ignores it.total_committed_recordscrates/feldera-types/src/coordination.rsIdle, so the report carries this step's countcrates/adapters/src/controller.rsopenapi.json,types.gen.ts)crates/feldera-types/src/adapter_stats.rs,crates/adapters/src/controller/stats.rsstep_status_reports_committed_recordscrates/adapters/src/controller/test.rsTested: the new test passes, and fails (
Some(0)instead ofSome(5)) if the host reports the count from before the step commits. The adapterscontroller::andserver::tests pass, exceptserver::test_with_kafka::test_server, which needs a Kafka broker. Not yet tested end to end with the coordinator change on a multihost cluster.