← 输入框|github.com|刷新
Skip to content

[adapters] Record the output assignment in each checkpoint. - #7942

Merged
blp merged 3 commits into
mainfrom
record-output-assignment
Nov 13, 2026
Merged

blp merged 3 commits into
mainfrom
record-output-assignment

Conversation

@blp

@blp blp commented Nov 11, 2026 •

Copy link
Copy Markdown
Member

A multihost pipeline keeps the state of each materialized view on the host that the coordinator assigns the view's stream to. Thus, the assignment must not change across a restart. Until now, nothing recorded the assignment, and the coordinator computed it again on each start. A change to the streams, to their connectors, or to the coordinator's cost estimates could move a view that did not change. The pipeline then failed to start with UnexpectedBootstrap.

Commit Change
Add an in-process multihost test harness MultihostTestPipeline runs all hosts in one process, so tests can checkpoint and restart them.
Record the output assignment in each checkpoint state.json gets an output_assignment field.
Report a checkpoint's output assignment to the coordinator New endpoint /coordination/checkpoint/output_assignment and capability checkpoint_output_assignment.

Endpoint replies:

Case Reply
Checkpoint has a record 200 with the assignment
Checkpoint predates the record 200 with no assignment
Host does not have the checkpoint 400
Host cannot read state.json 500

The coordinator side is feldera/cloud#2210. #7843 builds on this PR.

Limit: the first restart from a checkpoint that predates this change still computes the assignment again. The checkpoints that it writes have the record, so later restarts keep the assignment.

@blp blp added bug Something isn't working connectors Issues related to the adapters/connectors crate rust Pull requests that update Rust code multihost Related to multihost or distributed pipelines labels Nov 11, 2026
@blp blp self-assigned this Nov 11, 2026
@blp blp added connectors Issues related to the adapters/connectors crate rust Pull requests that update Rust code multihost Related to multihost or distributed pipelines labels Nov 11, 2026
@blp
blp requested a review from ryzhyk November 11, 2026 22:58
Comment thread crates/connectors/src/controller.rs Outdated
input_statistics,
output_statistics,
input_distribution: checkpointed_input_distribution,
output_assignment: _,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pitfall (medium): The host drops its own record here. If the coordinator sends an activation assignment that differs from the checkpoint's (an older coordinator, a coordinator bug, or a host that reports 500), an unchanged view still moves, and the user still sees the unclear UnexpectedBootstrap. Could you keep the record and compare it in with_output_assignment? A clear error (or at least a warn!) that names each moved stream with its old and new host would make that failure easy to diagnose. As it stands, the next checkpoint also overwrites the record without any sign.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done: the host now keeps the record and logs a warning for each stream that the activation moves away from the host in the record, with both hosts (moved_output_streams in controller.rs).

}
}
};
for (host, &checkpoint) in checkpoints.iter().enumerate() {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

missing test (medium): The doc comment and the commit message both say the endpoint works before activation, and the coordinator relies on that because it calls the endpoint before it builds CoordinationActivate. Every query in this test runs after start_with_storage has activated the hosts and they are running. Could you query at least one host after bootstrap but before /coordination/activate? Splitting the harness into start and activate steps would allow this. Two more cases have no test: a host without storage (400), and a single-host checkpoint, whose state.json should have no output_assignment key.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done: the harness can now start and activate the hosts in separate steps, and the restart test queries each host before activation. New tests cover a host without storage (400) and a single-host checkpoint (no output_assignment key).

@mythical-fred-oss

Copy link
Copy Markdown
Contributor

What I ran

Check Result
cargo check --locked -p dbsp_connectors ✅
cargo test --locked -p static-connectors multihost_restart_from_checkpoint ✅
New test 30× in a row, plus 3 rounds of 8 parallel copies (flakiness under load) ✅ 0 failures
All 38 server::test_http tests (checks that CIRCUIT_BUILD does not deadlock other tests) ✅
pre-commit run on the changed files ✅

Gates: The PR passes them. The integration test covers the new behaviour. No unsafe code, no new dependencies, no workflow changes. The coordination endpoint is internal, so no docs are needed. Checkpoint adds an optional field with serde(default) and has no deny_unknown_fields, so state.json stays readable by older and newer releases.

Not covered by tests (medium): see the inline comments.

  • The endpoint is never queried before activation, which is the case the coordinator relies on.
  • A host without storage, and a single-host checkpoint, have no test.
  • A restart where the activation's assignment differs from the checkpoint's has no test. This is the failure the PR exists to prevent, and nothing on the host side detects it.

Otherwise LGTM.

#[serde(default, skip_serializing_if = "BTreeMap::is_empty")]
pub input_distribution: BTreeMap<String, InputDistribution>,

/// The host that the coordinator assigned each output stream of a

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Don't you think that having an abstraction for a host could be useful?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good idea. A host is a usize in all of the coordination types today, on both sides, so a type for it is its own change. I filed feldera/cloud#2212.

/// The pipeline reports the output assignment that a checkpoint recorded
/// at `/coordination/checkpoint/output_assignment`.
#[serde(default)]
pub checkpoint_output_assignment: bool,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the name of this field does not suggest a boolean value

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Renamed to has_checkpoint_output_assignment.

Comment thread crates/static-connectors/src/engine.rs Outdated
}
}

/// Serializes the circuit builds of [static_circuit_factory]; see there.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"serializes" makes me think of serde

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reworded: "Lets [static_circuit_factory] build only one circuit at a time".

step: feldera_types::coordination::Step,
}

impl MultihostTestPipeline {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks like a useful tool. How come it didn't exist before?
Could it be reused by other tests?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is new in this PR. Before, the multihost tests in this repo ran one host with a multihost configuration, and the Python platform tests ran several hosts on Kubernetes. Nothing ran several hosts that exchange data in one process. Other tests can use it: #7843 uses it in about 20 tests.

Comment thread crates/connectors/src/server.rs Outdated
}

/// Returns the output assignment that the checkpoint named in the query
/// recorded, so that the coordinator can keep each output stream on its host

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no need to explain why. Just what is enough.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done: the comment now says only what the endpoint does.

Comment thread crates/connectors/src/server.rs Outdated
/// when it starts from that checkpoint. The host need not be activated.
///
/// The reply has no assignment if the checkpoint recorded none, because it
/// predates the record. A checkpoint that this host does not have, or whose

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't understand "a checkpoint that this host does not have".
Maybe "being unable to locate the checkpoint?"

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done: it now says "cannot find the checkpoint", and so does the error message.

Comment thread crates/connectors/src/server.rs Outdated
///
/// The reply has no assignment if the checkpoint recorded none, because it
/// predates the record. A checkpoint that this host does not have, or whose
/// record it cannot read, is an error, so that the coordinator does not take

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Again I find the "why" less useful.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done, removed.

@ryzhyk

ryzhyk commented Nov 12, 2026

Copy link
Copy Markdown
Contributor

There's apparently a corner case with case-sensitive table names:

The end-to-end probe confirms finding 3: a connector whose stream isn't in canonical form ends up on a host that gets none of the view's rows. I ran it on two in-process hosts with this PR's test harness, placing the connector the way the coordinator does, and sent 300 rows each time:

┌────────────────────┬─────────────────────────────────┬─────────────────────────────┐
│  Connector stream  │           Assignment            │     Rows in the host-0      │
│                    │                                 │      connector's file       │
├────────────────────┼─────────────────────────────────┼─────────────────────────────┤
│ test_output1       │ {"test_output1": 0}             │ 300                         │
│ (control)          │                                 │                             │
├────────────────────┼─────────────────────────────────┼─────────────────────────────┤
│ TEST_OUTPUT1       │ {"TEST_OUTPUT1": 0,             │ 0                           │
│                    │ "test_output1": 1}              │                             │
└────────────────────┴─────────────────────────────────┴─────────────────────────────┘

So a SQL connector on a view named with uppercase letters or quotes can produce nothing in a multihost pipeline, with no error. That happens whenever the coordinator puts the as-written name and the canonical name on different hosts, which it did for the simplest one-view, two-host program. This bug is older than both PRs; #7942's record keeps the split across restarts. The fix is to canonicalize config.stream in Pipelines::new in feldera/cloud.

blp added 3 commits November 11, 2026 17:09
MultihostTestPipeline runs the hosts of a multihost pipeline in one
process, with their exchange sockets bound ahead of time, and the test
acts as the coordinator, so tests can checkpoint and restart them.

Signed-off-by: Ben Pfaff <blp@feldera.com>
A multihost pipeline keeps the integral of each materialized view and
index on the workers of the host that the coordinator assigns its stream
to, so the assignment must not change across a restart.  Until now,
nothing recorded it.  The coordinator computed it afresh on every start,
from the streams, their output connectors, and its cost estimates, so a
change to any of these, or a coordinator release that estimates costs
differently, could move a view that did not change.  The restarted host
then found no state for the view and started to rebuild it, but the
pipeline diff knew nothing of the move and was empty, so the controller
refused to start with "UnexpectedBootstrap", and the pipeline could not
start until the assignment moved back.

A checkpoint now records the output assignment, and the next commit
reports it to the coordinator, so that the coordinator can keep each
stream on its host.  A restart from a checkpoint that predates this change
still computes the assignment afresh, but the checkpoints that it writes
record the assignment, so later restarts keep it.

If the coordinator's assignment moves a stream away from the host that
the checkpoint recorded, the host now logs a warning that names the
stream and both hosts, so that the failure that follows is easy to
diagnose.

Signed-off-by: Ben Pfaff <blp@feldera.com>
The new `/coordination/checkpoint/output_assignment` endpoint returns the
output assignment that a checkpoint recorded, and the new
`has_checkpoint_output_assignment` capability says that a host has it.  A host
answers before it is activated, so that the coordinator, when it starts
from a checkpoint, can keep each output stream on the host that holds its
state instead of moving the stream and rebuilding that state.

The reply has no assignment only if the checkpoint recorded none, because
it predates the record.  A checkpoint that the host cannot find, or whose
record the host cannot read, is an error, so that the coordinator does not
take it for an old checkpoint and move views.

The test harness can now start the hosts and activate them in separate
steps, so that a test can query a host before activation, as the
coordinator does.

Signed-off-by: Ben Pfaff <blp@feldera.com>
@blp
blp force-pushed the record-output-assignment branch from 8b16ed1 to 06b85c6 Compare November 12, 2026 04:50
@blp

blp commented Nov 12, 2026

Copy link
Copy Markdown
Member Author

@ryzhyk Thanks for the canonical name case. The fix is in feldera/cloud#2210, in the new commit "[coord] Place output connectors by the canonical names of their streams.": the coordinator now keys each connector by the canonical name of its stream, as the pipeline looks it up. Its test has connectors on v, V, "v", and "Mixed", and checks that the first three go to the same host.

@blp
blp added this pull request to the merge queue Nov 12, 2026
Merged via the queue into main with commit 5a59043 Nov 13, 2026
1 check passed
@blp
blp deleted the record-output-assignment branch November 13, 2026 00:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working connectors Issues related to the adapters/connectors crate multihost Related to multihost or distributed pipelines rust Pull requests that update Rust code

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants