Repository navigation
Spread view integrals over all hosts, build multihost snapshots, and add distributed output connectors - #7843
Conversation
| for part in &readers.partitions { | ||
| part.serialize_chunks(256, &mut |chunk| { | ||
| n_chunks += 1; | ||
| let batch = unsafe { handle.deserialize_chunk(chunk) }; |
There was a problem hiding this comment.
gate (high): missing SAFETY comment. Every unsafe block needs a // SAFETY: comment, test code included. Something like: // SAFETY: chunkwas just written byserialize_chunks of this same relation's integral.
missing (high): SQL type coverage. This is a new wire encoding (rkyv chunks sent between hosts), but the round trip only covers BIGINT, BOOLEAN and VARCHAR. Please run the same round trip over DeltaTestStruct or a schema that has every type: decimals, timestamps, intervals, binary, arrays, maps, structs, variant, nullables.
|
|
||
| let leases = TOKIO.block_on(async { | ||
| let mut leases = Vec::new(); | ||
| for host in (0..hosts.len()).filter(|&host| host != local_host) { |
There was a problem hiding this comment.
pitfall (medium): the circuit thread blocks on HTTP with no timeout. These lease calls run on the circuit thread, and PeerClient's reqwest client sets no connect_timeout and no timeout. If one peer's HTTP server accepts the connection but never answers, every step on this host stalls with no log. Please set a timeout on the client, or at least on lease. A slow lease then just means "try again at the next step", as it does today on errors.
| report.step | ||
| ); | ||
| } | ||
| Ok(response) => { |
There was a problem hiding this comment.
bug (medium): a part can retry forever, and the connector status never shows it. Every non-409 reply is retried with no limit. Here is one way it happens: the leader's endpoint is disconnected, so disconnect_output removes it from output_leaders. Every later report then gets 503 "no leader on this host yet". The follower's output thread spins on that for good, its queue fills, and backpressure stalls the pipeline. The only sign is a warn! at power-of-two attempts. The reverse case has the same shape: CommitLeader::gather waits forever for a follower part that died.
I checked this with a throwaway test: a server that always answers 503, and report_output was still retrying after 15 s.
Suggestion: once the leader has existed, treat "no leader" as fatal. Or bound the retries, and also put the stall into the connector's health/status, not only the log.
| /// Returns an error if output connector `config` sets `distributed` but its | ||
| /// transport does not support it (see | ||
| /// `TransportConfig::supports_output_distribution`). | ||
| pub fn validate_output_distribution( |
There was a problem hiding this comment.
breaking (medium): this validation now applies to every pipeline, including single-host ones.
- Before this PR,
distributedon an output was documented as "Valid for input connectors only" and was silently ignored. A pipeline that sets it on a Kafka, file, or other output now fails to start after the upgrade. - A single-host Delta output with
distributed: trueplusenable_output_bufferis also refused. That contradicts the docs, which say "In a single-host pipeline, this setting has no effect."
Please either skip these checks when the pipeline is not multihost, or list this change in the changelog as a behaviour change.
|
What I ran
Gates and higher-level issues
|
fe29ca7 to
02eb986
Compare
ryzhyk
left a comment
There was a problem hiding this comment.
Still working on it. See a couple of suggestions I have so far inline.
| /// | ||
| /// | Materialized relation | Registered by | Integral on multihost | Integral shard key | | ||
| /// |---|---|---|---| | ||
| /// | Table | [register_materialized_input_zset] | Each worker's own input, not sharded | | |
There was a problem hiding this comment.
More precisely it assumes the input is already sharded, which is the case for input streams produced by add_input_zset.
| /// rebuild it, and give the relation a second time to readers that | ||
| /// already have it. | ||
| #[track_caller] | ||
| pub fn output_accumulated_stream_following_with_gid<B>( |
There was a problem hiding this comment.
I think this is an overkill. This new mechanism makes sure that setting the distributed attribute doesn't lead to bootstrapping. I don't think it's an important case to cover and it's not worth the added complexity. I suggest we remove this mechanism and accept that (un)setting the distributed attribute can trigger bootstrapping.
There was a problem hiding this comment.
Done: removed in 3fdef33. Turning distributed on now starts a bootstrap, which sends the whole view to the connector again.
|
By examining individual features in this PR, claude found several serious issue. I will summarize them here, once comment per issue. Issue1: Any change that changes cost estimation causes bootstrapping. In addition, upgrading to the new runtime with this PR will cause bootstrapping, even though docs say it shouldn't; however I don't think that's an issue. |
|
Issue 2. This one I haven't tried to understand in detail, and just passing on as is I asked for a step-by-step scenario: |
|
Issue 3. A distributed input plus a distributed Delta output to a new table can never start |
|
Issue 4. Multihost snapshots are encoded one piece at a time (3; verified) |
|
Medium importance issues (some looks pretty high to me): |
|
02eb986 to
4fb7ade
Compare
3fdef33 to
b0f6c7b
Compare
|
@ryzhyk This PR is ready for re-review, together with feldera/cloud#2177. Here is where each issue from your review is now. Earlier replies give commit hashes, which later rebases changed, so this table names the commits by title.
Also, the mechanism that kept a view from being sent again when Each new Rust test fails with its fix reverted. The PR description has the full list of commits and changes. Merge order: #7942, then feldera/cloud#2210, then this PR, then feldera/cloud#2177. Each cloud PR first moves its submodule to the feldera |
Multihost pipelines gather output delta streams, and the integrals of materialized views, to the host that the coordinator assigns to each stream, but tables spread their integrals over all workers. Add a table to the documentation of Catalog that shows each case and explains why, so that later changes to output placement can update it. Signed-off-by: Ben Pfaff <blp@feldera.com>
| /// | ||
| /// After each warning interval of waiting, logs a warning and calls `on_wait` with how long it | ||
| /// has waited and the hosts that have reported. | ||
| pub fn gather( |
There was a problem hiding this comment.
bug (medium): leader and follower can deadlock when the follower is behind. gather handles a part that skipped ahead (it checks range(step + 1..)), but not one that is still waiting on an earlier step. If host 1 has a pending report of step 8 and the leader gathers step 10, then:
- host 1 blocks in
report(8)and never reports 10; finish()would fail step 8, but it only runs after step 10 completes.
So neither side moves. Only shutdown ends it, and the connector turns unhealthy after 10 min. This can happen when the leader is recreated after disconnect_output, because output_part replaces the closed leader while a follower is still retrying an older step. A connector added through the API at different steps on different hosts can hit it too. Repro (fails; gather is still waiting after 3 s):
let leader = Arc::new(CommitLeader::new(2));
let rt = tokio::runtime::Runtime::new().unwrap();
let _behind = rt.spawn({ let l = leader.clone(); async move { l.report(1, 8, json!(1)).await } });
std::thread::sleep(Duration::from_millis(100));
let g = std::thread::spawn({ let l = leader.clone(); move || l.gather(10, &CancellationToken::new(), &mut |_, _| {}) });
std::thread::sleep(Duration::from_secs(3));
assert!(g.is_finished());Suggested fix: at the start of gather(step), fail every pending step < step the same way finish does, so the waiting parts get an error and can report later steps.
There was a problem hiding this comment.
Fixed in 1c9b830, "[adapters] Fail a distributed commit report of a step that the leader passed": gather(step) now fails every pending report of an earlier step, as finish does, so the part gets an error and goes on to report the later step. Your repro is the new test a_part_that_is_behind_fails_instead_of_waiting, which also checks that the part's next report completes the gather. It fails without the fix.
| decoder.push(&response.body().await.unwrap()); | ||
| let mut chunks = 0; | ||
| while let Some(chunk) = decoder.next_chunk() { | ||
| rows += unsafe { handle.deserialize_chunk(&chunk) }.len(); |
There was a problem hiding this comment.
missing (low, hard-rule gate): no // SAFETY: comment on this unsafe block. It's test code, but the rule covers every unsafe block. Copy the comment from static-connectors/src/catalog.rs, e.g. // SAFETY: this host's own scan of this relation's integral wrote chunk.
There was a problem hiding this comment.
Added, in "[adapters] Build multihost snapshots from the parts on every host", the commit that added this test.
|
What I ran (debug builds, on the merge commit)
Gates: tests, docs and changelog, SQL type coverage for chunks, and the bug (medium): the commit leader deadlocks when a follower is still waiting on an earlier step than the one the leader gathers (see the inline comment and its repro). I'd fix this before merge, plus a unit test for it. missing tests (medium):
|
62eb86c to
209c7ae
Compare
|
Thanks for the QA pass. Pushed as 209c7ae:
|
ryzhyk
left a comment
There was a problem hiding this comment.
Thanks for addressing everything!
In a multihost pipeline, the integral of each materialized view was gathered onto the host that the coordinator assigned to its stream. That host held the whole view, and its workers did all of the view's work. The integral was there only so that a `send_snapshot` connector on that host could read the whole view. Ad hoc queries do not need it: they read each worker's part of an integral on that worker's host. With this change, a new multihost pipeline shards the integral of each materialized view and index over all workers of all hosts, so that a large view's state and work spread over all of them. No host then has the whole view, so a multihost pipeline no longer supports `send_snapshot`. It refuses an output connector that sets it, and HTTP egress refuses `send_snapshot=true`, whatever the layout of the integrals. An ad hoc query reads the whole view instead. A checkpoint now records where the view integrals are, and a pipeline that restarts from a checkpoint keeps that layout, so that an upgrade does not move its state. A checkpoint from before this change keeps them on one host. The new `dev_tweaks.view_integrals` setting overrides the choice: `one_host` keeps the old layout, in case the new one has a bug, and `all_hosts` moves a pipeline to the new one. The persistent id of an integral includes its range of workers, so a move in either direction finds no state and rebuilds the integrals as a bootstrap, which by default waits for approval, and the pipeline diff says why. With the integrals on all hosts, a view's state does not depend on its host, so a change of output assignment moves no state. The multihost tests check the layout that each checkpoint records, each layout switch, and the refusal of `send_snapshot`. They count the rows of the view on each host, as an ad hoc query reads them. Signed-off-by: Ben Pfaff <blp@feldera.com>
An output connector with `distributed: true` now runs on every host of a multihost pipeline, and each host's part writes the output of that host's workers. The null connector and the Delta Lake connector, with `mode: append` and `update_mode: cdc`, support it. On other transports, it has no effect, as before: the connector runs on one host, and the pipeline logs a warning. A distributed Delta Lake output still commits each transaction once. Each part writes its own data files and reports them, for each committed step, to the part on the stream's assigned host, which commits every part's files in one Delta commit and then answers the parts. Only that part creates the table. It commits only files inside the table. The other parts wait for the table to exist. Output buffering is refused for now. The parts call the leader over HTTPS, at the base URL of each host, which the coordinator now sends in `CoordinationActivate::host_urls`. When the pipeline does not require client certificates, each host trusts the root CA of its own pipeline certificate chain, as the coordinator does. Without host URLs, as from an older coordinator, a part that is not the leader refuses to start. A part's report lists every file that it wrote in a step, so the leader takes reports of up to 256 MiB, instead of actix's default 2 MiB limit for JSON. A part retries a report while the leader is busy, but not one that the leader refuses, for example as too large, because the leader would refuse it again. A part of a distributed Delta Lake output that is not the leader waits, as it initializes, for the leader to create the table. When an input connector needs host 0's choice, the coordinator activates host 0 alone and activates the other hosts only after host 0 publishes its choices. Host 0 published them only after all of its connectors initialized, so with the leader on another host, host 0's part waited for a table that nothing could create, and the pipeline could never start. Two changes fix this: - A part that is not the leader no longer waits for the table while it initializes. It opens the table before it writes its first files, in a step, and the pipeline runs its first step only after the coordinator has activated every host. The connector traits now state this rule: a connector must not wait for another host while it initializes. - A host of a multihost pipeline initializes its input connectors, publishes their choices, and only then initializes its output connectors, so that an output connector that breaks the rule cannot hold back the other hosts. The in-process multihost test harness can run 3 hosts, passes each host the other hosts' URLs, and can activate one host at a time, as the coordinator does. The connector's status combines the statuses of its parts. The distributed output of a relation is an output operator of its own, so a restart that turns `distributed` on for a connector bootstraps that operator, which sends the whole view to the connector again. The docs describe distributed output connectors, and the Python platform tests check a distributed Delta Lake output's deltas and resume. Signed-off-by: Ben Pfaff <blp@feldera.com>
A restart that changes where a multihost pipeline keeps the state of its materialized views (`dev_tweaks.view_integrals`) rebuilds the state of every view. A concurrent bootstrap leaves the views that it rebuilds out of the live circuit until cutover, so every view would read as empty to ad hoc queries until the rebuild ended. Such a restart now fails, as other changes that cannot be bootstrapped concurrently do, with advice to restart with `concurrent_bootstrap=false`. Signed-off-by: Ben Pfaff <blp@feldera.com>
…ut host URLs. A coordinator that predates `CoordinationActivate::host_urls` sends no host URLs. The parts of a distributed Delta Lake output that are not the leader refused to start without them, but the leader started, and then waited forever for the reports of parts that could not send them. Now every part refuses to start, the leader too. Signed-off-by: Ben Pfaff <blp@feldera.com>
…ealthy. The leader of a distributed Delta Lake output marked the connector unhealthy after it had waited a minute for the other parts to report a step. The parts report a step only once they have written all of it, so a large step on a healthy connector made it unhealthy. Now the connector stays healthy, with a description of the wait, until it has waited 10 minutes. The log still warns every minute. Also, `CommitLeader::gather` did not notice `close()`, so a leader whose connector stopped kept waiting. Now it stops. Signed-off-by: Ben Pfaff <blp@feldera.com>
The leader of a distributed Delta Lake output waits for every other part to report a step. A part that reported a later step without that one had skipped it, and it waited for its later step, which the leader would reach only after the step that it waited for, so the two waited for each other forever. Now the leader fails the step, and names the hosts that skipped it, so that the connector reports an error. Signed-off-by: Ben Pfaff <blp@feldera.com>
…need it. A distributed output connector puts the stream that it reads, and its index if it reads one, into the set of streams that get a change stream sharded over all workers. But the catalog checked an index by its view's name, because it registers an index with its view's schema. So the index names in the set were never used, and a distributed connector on a view gave the view and every index of the view a distributed change stream, each with an exchange of its own. A connector that read an index got its stream only because its view's name was in the set. Now the catalog checks an index by its own name, a view by its name and by the name of the index that it doubles as, and the set holds a connector's index if it reads one, and its stream otherwise. Signed-off-by: Ben Pfaff <blp@feldera.com>
Each host writes `__feldera_ts` from its own clock, so the filter on its maximum in the `MERGE INTO` example can skip rows if the hosts' clocks differ. And a distributed output needs a coordinator and pipeline hosts from the same release. Signed-off-by: Ben Pfaff <blp@feldera.com>
… passed. The commit leader of a distributed output failed a part that had skipped ahead of the step that it gathered, but not a part that was behind it. If a part waited for the answer to its report of step 8 while the leader gathered step 10, the part never reported step 10, and the leader never finished step 10, which was the only place that failed step 8. Both waited until shutdown, and the connector became unhealthy after 10 minutes. This can happen when the leader is recreated, after its connector is disconnected, while a part retries a report, or when hosts add the connector at different steps. Now gather fails every pending report of an earlier step, as finish already did, so that the part gets an error and reports the later step. Signed-off-by: Ben Pfaff <blp@feldera.com>
A report holds the Delta Lake `Add` action of each file that a part wrote in a step, with the file's column statistics, not only a list of files. Each entry is about 1-4 KB, so one large step can exceed actix's 2 MiB JSON limit. Signed-off-by: Ben Pfaff <blp@feldera.com>
209c7ae to
577ea84
Compare
|
Force-pushed as 577ea84: a multihost pipeline no longer supports
The commits after "Add distributed output connectors" are as before. #7957 is closed, because there are no snapshot chunks. |
…ite. When a part of a distributed Delta Lake output failed to write a batch, `encode` dropped the part's files of the step and returned an error. `steps_processed` then reported the files that the part still had, often none, and the leader committed every other part's files. A reader saw part of the transaction, with no error on the other hosts. This broke the promise in delta.md that a reader sees all of a transaction's output or none of it, as with one host, where a failed batch drops the whole step. Now a part that fails to write in `encode` reports the failure, in the new `error` field of `OutputPartReport`, instead of its files. If any part's report is a failure, the leader's own included, the leader commits none of the step, and every part gets an error that names the part that failed. The files of the step are orphans that `VACUUM` collects. This covers a part that cannot open the leader's table, too. Such a part does not stop, as a comment said: it tries again for each batch. Now each of its steps fails on every part, instead of committing without its share. The coordinator and the pipeline must be from the same release, as delta.md already says, so the new field needs no compatibility. New tests: `a_failed_part_fails_the_step`, for the leader, and `a_failed_part_commits_none_of_the_step`, for a Delta Lake part that fails one batch of a step. Each fails with its half of the fix reverted. Signed-off-by: Ben Pfaff <blp@feldera.com>
…uts found. - A restart warned that each output stream that the coordinator moved from its recorded host has its state rebuilt. With view integrals on all hosts, the new default, no host has a stream's state, so the warning was false, and it could make an operator delay a safe restart. Now it warns only when the integrals are on one host. - A part that could not open its leader's table reported the error as a fatal transport error, although it tries again for its next batch, and `encode` returned the same error, which the output thread reported a second time. Now only the output thread reports it, once. - A part reached its leader by indexing the host URLs that the coordinator sent, so a coordinator that sent too few would make the output thread panic. Now the controller refuses to start unless there is one URL for each host, and a report to a host with no URL is an error. New test: `check_host_urls`, which fails without the check. Signed-off-by: Ben Pfaff <blp@feldera.com>
In a multihost pipeline, the integral of each materialized view was gathered onto the one host that runs the view's output connectors. That host held the whole view and did all of its work. This PR spreads view integrals over all hosts and adds distributed output connectors, which run a part on every host.
With the integrals spread, no host has a whole view, so a multihost pipeline no longer supports
send_snapshot, on a configured connector or on HTTP egress. An ad hoc query reads the whole view instead. An earlier version of this PR built snapshots from the parts on every host. That was most of the PR's code and most of its review findings, so it is gone.This PR builds on #7942, which records the output assignment in each checkpoint and is merged.
Commits
Catalogdocs, so that the next commits show what changessend_snapshot. A checkpoint records where its view integrals are, and a restart keeps that layout. Addsdev_tweaks.view_integrals.distributed: trueon the Null and Delta Lake (mode: append,update_mode: cdc) output connectors. A distributed Delta Lake output commits each transaction once, through a leader, which the other parts call at the host URLs that the coordinator sends. On other output connectors,distributed: truestill has no effect, and the pipeline logs a warning.dev_tweaks.view_integralsrebuilds every view, and a concurrent bootstrap would leave every view empty until cutover, so such a restart fails with advice to restart withconcurrent_bootstrap=false.gatherstops onclose().__feldera_tscomes from each host's clock, and the coordinator and pipeline must be from the same release.Addaction, with column statistics.Each commit builds on its own.
Behavior changes
Incompatible (multihost):
send_snapshotis refused. A pipeline with an output connector that sets it fails to start, and HTTP egress refusessend_snapshot=true. The changelog has an entry.Multihost: a new pipeline spreads the integrals of its materialized views over all hosts. A pipeline that restarts from a checkpoint keeps the checkpoint's layout, so the upgrade moves no state. The changelog has an entry.
dev_tweaks.view_integralsoverrides the layout:auto(default)one_hostall_hostsA move in either direction rebuilds the integrals as a bootstrap. With the default bootstrap policy,
await_approval, it waits for approval, and the pipeline diff says why.Notes for review
Rollout
The runtime needs the coordinator from feldera/cloud#2177, which sends each host the URLs of the other hosts. Without them, a distributed Delta Lake output refuses to start. Release both together.
Measurement
A 4-host QA workload (8 workers per host) joins a 1.07B-row CDC table with a 64M-row table into a materialized view, then computes a 90-day rolling aggregate. It loads 40 days of input, one transaction per day. Two runs differ only in
dev_tweaks.view_integrals:one_hostall_hosts(new default)The spread layout removes about 70% of the cost of the materialized view. The last column is an earlier run on an older runtime, for reference. These runs used the earlier version of this PR, whose layout is the same.
Testing
send_snapshot, restarts from checkpoints and layout switches, which count each host's rows of a view as an ad hoc query reads them, distributed Null and Delta Lake output, commit failures, staggered connector starts, and the leader's report handling. Each new test fails with its fix reverted.test_delta_output_distributed.py, which CI'spython-multihostsuite runs on 2 hosts.Follow-ups
Catalog::gather_output_to_hostcould return a struct instead of a 4-tuple./coordination/*to the pipeline's coordinator and hosts: Limit the /coordination/* endpoints to the pipeline's coordinator and hosts #7958.The coordinator changes are in feldera/cloud#2177, which needs this PR.