Repository navigation
[adapters] Record the output assignment in each checkpoint. - #7942
Conversation
| input_statistics, | ||
| output_statistics, | ||
| input_distribution: checkpointed_input_distribution, | ||
| output_assignment: _, |
There was a problem hiding this comment.
pitfall (medium): The host drops its own record here. If the coordinator sends an activation assignment that differs from the checkpoint's (an older coordinator, a coordinator bug, or a host that reports 500), an unchanged view still moves, and the user still sees the unclear UnexpectedBootstrap. Could you keep the record and compare it in with_output_assignment? A clear error (or at least a warn!) that names each moved stream with its old and new host would make that failure easy to diagnose. As it stands, the next checkpoint also overwrites the record without any sign.
There was a problem hiding this comment.
Done: the host now keeps the record and logs a warning for each stream that the activation moves away from the host in the record, with both hosts (moved_output_streams in controller.rs).
| } | ||
| } | ||
| }; | ||
| for (host, &checkpoint) in checkpoints.iter().enumerate() { |
There was a problem hiding this comment.
missing test (medium): The doc comment and the commit message both say the endpoint works before activation, and the coordinator relies on that because it calls the endpoint before it builds CoordinationActivate. Every query in this test runs after start_with_storage has activated the hosts and they are running. Could you query at least one host after bootstrap but before /coordination/activate? Splitting the harness into start and activate steps would allow this. Two more cases have no test: a host without storage (400), and a single-host checkpoint, whose state.json should have no output_assignment key.
There was a problem hiding this comment.
Done: the harness can now start and activate the hosts in separate steps, and the restart test queries each host before activation. New tests cover a host without storage (400) and a single-host checkpoint (no output_assignment key).
|
What I ran
Gates: The PR passes them. The integration test covers the new behaviour. No unsafe code, no new dependencies, no workflow changes. The coordination endpoint is internal, so no docs are needed. Not covered by tests (medium): see the inline comments.
Otherwise LGTM. |
| #[serde(default, skip_serializing_if = "BTreeMap::is_empty")] | ||
| pub input_distribution: BTreeMap<String, InputDistribution>, | ||
|
|
||
| /// The host that the coordinator assigned each output stream of a |
There was a problem hiding this comment.
Don't you think that having an abstraction for a host could be useful?
There was a problem hiding this comment.
Good idea. A host is a usize in all of the coordination types today, on both sides, so a type for it is its own change. I filed feldera/cloud#2212.
| /// The pipeline reports the output assignment that a checkpoint recorded | ||
| /// at `/coordination/checkpoint/output_assignment`. | ||
| #[serde(default)] | ||
| pub checkpoint_output_assignment: bool, |
There was a problem hiding this comment.
the name of this field does not suggest a boolean value
There was a problem hiding this comment.
Renamed to has_checkpoint_output_assignment.
| } | ||
| } | ||
|
|
||
| /// Serializes the circuit builds of [static_circuit_factory]; see there. |
There was a problem hiding this comment.
"serializes" makes me think of serde
There was a problem hiding this comment.
Reworded: "Lets [static_circuit_factory] build only one circuit at a time".
| step: feldera_types::coordination::Step, | ||
| } | ||
|
|
||
| impl MultihostTestPipeline { |
There was a problem hiding this comment.
This looks like a useful tool. How come it didn't exist before?
Could it be reused by other tests?
There was a problem hiding this comment.
It is new in this PR. Before, the multihost tests in this repo ran one host with a multihost configuration, and the Python platform tests ran several hosts on Kubernetes. Nothing ran several hosts that exchange data in one process. Other tests can use it: #7843 uses it in about 20 tests.
| } | ||
|
|
||
| /// Returns the output assignment that the checkpoint named in the query | ||
| /// recorded, so that the coordinator can keep each output stream on its host |
There was a problem hiding this comment.
no need to explain why. Just what is enough.
There was a problem hiding this comment.
Done: the comment now says only what the endpoint does.
| /// when it starts from that checkpoint. The host need not be activated. | ||
| /// | ||
| /// The reply has no assignment if the checkpoint recorded none, because it | ||
| /// predates the record. A checkpoint that this host does not have, or whose |
There was a problem hiding this comment.
I don't understand "a checkpoint that this host does not have".
Maybe "being unable to locate the checkpoint?"
There was a problem hiding this comment.
Done: it now says "cannot find the checkpoint", and so does the error message.
| /// | ||
| /// The reply has no assignment if the checkpoint recorded none, because it | ||
| /// predates the record. A checkpoint that this host does not have, or whose | ||
| /// record it cannot read, is an error, so that the coordinator does not take |
There was a problem hiding this comment.
Again I find the "why" less useful.
|
There's apparently a corner case with case-sensitive table names: |
MultihostTestPipeline runs the hosts of a multihost pipeline in one process, with their exchange sockets bound ahead of time, and the test acts as the coordinator, so tests can checkpoint and restart them. Signed-off-by: Ben Pfaff <blp@feldera.com>
A multihost pipeline keeps the integral of each materialized view and index on the workers of the host that the coordinator assigns its stream to, so the assignment must not change across a restart. Until now, nothing recorded it. The coordinator computed it afresh on every start, from the streams, their output connectors, and its cost estimates, so a change to any of these, or a coordinator release that estimates costs differently, could move a view that did not change. The restarted host then found no state for the view and started to rebuild it, but the pipeline diff knew nothing of the move and was empty, so the controller refused to start with "UnexpectedBootstrap", and the pipeline could not start until the assignment moved back. A checkpoint now records the output assignment, and the next commit reports it to the coordinator, so that the coordinator can keep each stream on its host. A restart from a checkpoint that predates this change still computes the assignment afresh, but the checkpoints that it writes record the assignment, so later restarts keep it. If the coordinator's assignment moves a stream away from the host that the checkpoint recorded, the host now logs a warning that names the stream and both hosts, so that the failure that follows is easy to diagnose. Signed-off-by: Ben Pfaff <blp@feldera.com>
The new `/coordination/checkpoint/output_assignment` endpoint returns the output assignment that a checkpoint recorded, and the new `has_checkpoint_output_assignment` capability says that a host has it. A host answers before it is activated, so that the coordinator, when it starts from a checkpoint, can keep each output stream on the host that holds its state instead of moving the stream and rebuilding that state. The reply has no assignment only if the checkpoint recorded none, because it predates the record. A checkpoint that the host cannot find, or whose record the host cannot read, is an error, so that the coordinator does not take it for an old checkpoint and move views. The test harness can now start the hosts and activate them in separate steps, so that a test can query a host before activation, as the coordinator does. Signed-off-by: Ben Pfaff <blp@feldera.com>
8b16ed1 to
06b85c6
Compare
|
@ryzhyk Thanks for the canonical name case. The fix is in feldera/cloud#2210, in the new commit "[coord] Place output connectors by the canonical names of their streams.": the coordinator now keys each connector by the canonical name of its stream, as the pipeline looks it up. Its test has connectors on |
A multihost pipeline keeps the state of each materialized view on the host that the coordinator assigns the view's stream to. Thus, the assignment must not change across a restart. Until now, nothing recorded the assignment, and the coordinator computed it again on each start. A change to the streams, to their connectors, or to the coordinator's cost estimates could move a view that did not change. The pipeline then failed to start with
UnexpectedBootstrap.MultihostTestPipelineruns all hosts in one process, so tests can checkpoint and restart them.state.jsongets anoutput_assignmentfield./coordination/checkpoint/output_assignmentand capabilitycheckpoint_output_assignment.Endpoint replies:
state.jsonThe coordinator side is feldera/cloud#2210. #7843 builds on this PR.
Limit: the first restart from a checkpoint that predates this change still computes the assignment again. The checkpoints that it writes have the record, so later restarts keep the assignment.