← 输入框|github.com|刷新
Skip to content

Spread view integrals over all hosts, build multihost snapshots, and add distributed output connectors - #7843

Merged
ryzhyk merged 13 commits into
mainfrom
distributed-outputs
Nov 14, 2026
Merged

ryzhyk merged 13 commits into
mainfrom
distributed-outputs

Conversation

@blp

@blp blp commented Nov 4, 2026 •

Copy link
Copy Markdown
Member

In a multihost pipeline, the integral of each materialized view was gathered onto the one host that runs the view's output connectors. That host held the whole view and did all of its work. This PR spreads view integrals over all hosts and adds distributed output connectors, which run a part on every host.

With the integrals spread, no host has a whole view, so a multihost pipeline no longer supports send_snapshot, on a configured connector or on HTTP egress. An ad hoc query reads the whole view instead. An earlier version of this PR built snapshots from the parts on every host. That was most of the PR's code and most of its review findings, so it is gone.

This PR builds on #7942, which records the output assignment in each checkpoint and is merged.

Commits

Commit What it does
Document which workers hold each output stream and integral Documents the layout before this PR, in the Catalog docs, so that the next commits show what changes
Spread materialized view integrals over all hosts A new multihost pipeline shards view integrals over all workers of all hosts. A multihost pipeline refuses send_snapshot. A checkpoint records where its view integrals are, and a restart keeps that layout. Adds dev_tweaks.view_integrals.
Add distributed output connectors distributed: true on the Null and Delta Lake (mode: append, update_mode: cdc) output connectors. A distributed Delta Lake output commits each transaction once, through a leader, which the other parts call at the host URLs that the coordinator sends. On other output connectors, distributed: true still has no effect, and the pipeline logs a warning.
Refuse a concurrent bootstrap of a change of view layout A restart that changes dev_tweaks.view_integrals rebuilds every view, and a concurrent bootstrap would leave every view empty until cutover, so such a restart fails with advice to restart with concurrent_bootstrap=false.
Refuse a distributed Delta Lake output on every host without host URLs The leader, too, refuses to start without host URLs, instead of waiting forever for reports.
Do not report a slow but healthy distributed commit as unhealthy A leader that waits for the other parts stays healthy, with a description, until 10 minutes; gather stops on close().
Fail a distributed commit whose part skipped the step A part that reported a later step without the one that the leader waits for fails that step, instead of both waiting forever.
Give a distributed change stream only to the streams that need it An index goes by its own name for its distributed change stream, so a distributed connector no longer gives its view and every index of the view one.
Note two limits of distributed Delta Lake output Docs: __feldera_ts comes from each host's clock, and the coordinator and pipeline must be from the same release.
Fail a distributed commit report of a step that the leader passed A part that waits on an earlier step gets an error when the leader gathers a later one, instead of both waiting until shutdown.
Say what makes a distributed commit report large A comment: a report holds each file's Delta Lake Add action, with column statistics.
Commit none of a distributed step that a part failed to write A part that fails to write its output reports the failure, and the leader then commits none of the step, so a reader never sees part of a transaction.
Fix three small problems that a review of distributed outputs found No false warning about a moved stream when view integrals are on all hosts; a failure to open the leader's table is reported once and not as fatal; the controller refuses a list of host URLs that is not one per host.

Each commit builds on its own.

Behavior changes

  • Incompatible (multihost): send_snapshot is refused. A pipeline with an output connector that sets it fails to start, and HTTP egress refuses send_snapshot=true. The changelog has an entry.

  • Multihost: a new pipeline spreads the integrals of its materialized views over all hosts. A pipeline that restarts from a checkpoint keeps the checkpoint's layout, so the upgrade moves no state. The changelog has an entry.

  • dev_tweaks.view_integrals overrides the layout:

    Value Layout
    auto (default) The checkpoint's layout. All hosts for a new pipeline. One host for a checkpoint from before this PR.
    one_host One host, as before this PR. For use if the new layout has a bug.
    all_hosts All hosts. Moves an existing pipeline to the new layout.

    A move in either direction rebuilds the integrals as a bootstrap. With the default bootstrap policy, await_approval, it waits for approval, and the pipeline diff says why.

Notes for review

  • Calls between hosts use HTTPS. Without client certificates, each host trusts the root CA of its own pipeline certificate chain, as the coordinator does. The only call is a part's report to its leader.
  • The leader protocol for distributed Delta Lake output: each part writes its own data files and reports them for each committed step to the part on the stream's assigned host, which commits all parts' files in one Delta commit. If any part fails to write its files for the step, the leader commits none of them. Output buffering is refused for distributed Delta Lake output for now.

Rollout

The runtime needs the coordinator from feldera/cloud#2177, which sends each host the URLs of the other hosts. Without them, a distributed Delta Lake output refuses to start. Release both together.

Measurement

A 4-host QA workload (8 workers per host) joins a 1.07B-row CDC table with a 64M-row table into a materialized view, then computes a 90-day rolling aggregate. It loads 40 days of input, one transaction per day. Two runs differ only in dev_tweaks.view_integrals:

one_host all_hosts (new default) View not materialized
Pipeline uptime 4789 s 4031 s 3701 s
Seconds per input day, days 31-40 160 104 92
Storage per host at the end, hosts 0/1/2/3 96 / 220 / 98 / 98 GB 111 / 110 / 110 / 111 GB
Fullest host against the mean, at the end 1.72 1.005
Peak storage, all hosts 515 GB 449 GB 347 GB

The spread layout removes about 70% of the cost of the materialized view. The last column is an earlier run on an older runtime, for reference. These runs used the earlier version of this PR, whose layout is the same.

Testing

  • Rust: in-process tests on 2 and 3 hosts for the refusal of send_snapshot, restarts from checkpoints and layout switches, which count each host's rows of a view as an ad hoc query reads them, distributed Null and Delta Lake output, commit failures, staggered connector starts, and the leader's report handling. Each new test fails with its fix reverted.
  • Python platform tests: test_delta_output_distributed.py, which CI's python-multihost suite runs on 2 hosts.

Follow-ups

  • Output buffering for distributed Delta Lake output: the leader could choose the flush points for all parts.
  • Catalog::gather_output_to_host could return a struct instead of a 4-tuple.
  • Limit /coordination/* to the pipeline's coordinator and hosts: Limit the /coordination/* endpoints to the pipeline's coordinator and hosts #7958.
  • A test that stops the commit leader's host in the middle of a commit, and checks recovery.
  • A part that cannot open the leader's table tries again on each step, for about 75 s each time, which delays the leader. Each such step now fails on every part, with an error, but the wait stays. The part could wait between tries, or stop.
  • Placement in the coordinator could count the integrals of a pipeline that keeps them on one host.

The coordinator changes are in feldera/cloud#2177, which needs this PR.

@blp blp added documentation Improvements or additions to documentation performance connectors Issues related to the adapters/connectors crate User-facing For PRs that lead to Feldera-user visible changes rust Pull requests that update Rust code python Pull requests that update python code enterprise Issue related to Feldera Enterprise features. multihost Related to multihost or distributed pipelines labels Nov 4, 2026
@blp blp self-assigned this Nov 4, 2026
@blp
blp requested a review from ryzhyk November 4, 2026 22:06
Comment thread crates/static-connectors/src/catalog.rs Outdated
for part in &readers.partitions {
part.serialize_chunks(256, &mut |chunk| {
n_chunks += 1;
let batch = unsafe { handle.deserialize_chunk(chunk) };

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

gate (high): missing SAFETY comment. Every unsafe block needs a // SAFETY: comment, test code included. Something like: // SAFETY: chunkwas just written byserialize_chunks of this same relation's integral.

missing (high): SQL type coverage. This is a new wire encoding (rkyv chunks sent between hosts), but the round trip only covers BIGINT, BOOLEAN and VARCHAR. Please run the same round trip over DeltaTestStruct or a schema that has every type: decimals, timestamps, intervals, binary, arrays, maps, structs, variant, nullables.

Comment thread crates/connectors/src/controller.rs Outdated

let leases = TOKIO.block_on(async {
let mut leases = Vec::new();
for host in (0..hosts.len()).filter(|&host| host != local_host) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pitfall (medium): the circuit thread blocks on HTTP with no timeout. These lease calls run on the circuit thread, and PeerClient's reqwest client sets no connect_timeout and no timeout. If one peer's HTTP server accepts the connection but never answers, every step on this host stalls with no log. Please set a timeout on the client, or at least on lease. A slow lease then just means "try again at the next step", as it does today on errors.

report.step
);
}
Ok(response) => {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bug (medium): a part can retry forever, and the connector status never shows it. Every non-409 reply is retried with no limit. Here is one way it happens: the leader's endpoint is disconnected, so disconnect_output removes it from output_leaders. Every later report then gets 503 "no leader on this host yet". The follower's output thread spins on that for good, its queue fills, and backpressure stalls the pipeline. The only sign is a warn! at power-of-two attempts. The reverse case has the same shape: CommitLeader::gather waits forever for a follower part that died.

I checked this with a throwaway test: a server that always answers 503, and report_output was still retrying after 15 s.

Suggestion: once the leader has existed, treat "no leader" as fatal. Or bound the retries, and also put the stall into the connector's health/status, not only the log.

/// Returns an error if output connector `config` sets `distributed` but its
/// transport does not support it (see
/// `TransportConfig::supports_output_distribution`).
pub fn validate_output_distribution(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

breaking (medium): this validation now applies to every pipeline, including single-host ones.

  • Before this PR, distributed on an output was documented as "Valid for input connectors only" and was silently ignored. A pipeline that sets it on a Kafka, file, or other output now fails to start after the upgrade.
  • A single-host Delta output with distributed: true plus enable_output_buffer is also refused. That contradicts the docs, which say "In a single-host pipeline, this setting has no effect."

Please either skip these checks when the pipeline is not multihost, or list this change in the changelog as a behaviour change.

@mythical-fred-oss

Copy link
Copy Markdown
Contributor

What I ran

Check Result
cargo check --locked -p dbsp_connectors --tests ✅
cargo test -p dbsp_connectors --lib (output_leader, snapshot_parts, distribution, part_file) ✅ 15/15
cargo test -p static-connectors --test connector_suites -- multihost snapshot_lease, run 5 times ✅ 15/15 every run, no flakes
cargo test -p static-connectors --lib (spreads / round_trips) ✅
pre-commit run --files <changed> ✅
QA: a throwaway test where the leader always answers 503 ❌ report_output never returns (see inline comment)

Gates and higher-level issues

  1. Gate, high: unsafe without a SAFETY comment. The unsafe block in the test integral_round_trips_through_chunks has no // SAFETY: comment (see inline comment).
  2. Gate, high: SQL type coverage. The new rkyv chunk path between hosts and the distributed Delta output are tested only with BIGINT, BOOLEAN and VARCHAR, in Rust and in the Python tests. Please add an all-types schema such as DeltaTestStruct, for both the snapshot round trip and distributed Delta output.
  3. Pitfall, high: no feature flag, and a costly upgrade. Spreading view integrals is now unconditional for every multihost pipeline.
    • After the upgrade, a pipeline restarting from an older checkpoint needs a bootstrap, and by default that bootstrap waits for approval.
    • Pipelines whose views read non-materialized tables cannot restart from their checkpoint at all, so they lose their state.
    • Please gate the new layout behind a dev_tweaks flag, or keep the OneHost layout when restoring an older checkpoint, until the new layout has had some soak time.
  4. Pitfall, medium: the runtime needs a new coordinator first. Multihost send_snapshot, including HTTP egress with a snapshot, is now refused unless the coordinator sends host_urls. A runtime that ships before that coordinator change regresses those connectors, so the rollout order matters.
  5. Missing, medium: untested failure cases.
    • A follower part, or the leader part, disconnects at runtime while the other parts keep reporting. Today this hangs (see inline comment).
    • The leader's host starts the connector later than a follower; the staggered test only has a follower start first.
    • A distributed Delta output resumes after the leader crashes between the Delta commit and its answer to the parts.

@blp
blp force-pushed the distributed-outputs branch from fe29ca7 to 02eb986 Compare November 5, 2026 16:12

@ryzhyk ryzhyk left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Still working on it. See a couple of suggestions I have so far inline.

///
/// | Materialized relation | Registered by | Integral on multihost | Integral shard key |
/// |---|---|---|---|
/// | Table | [register_materialized_input_zset] | Each worker's own input, not sharded | |

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

More precisely it assumes the input is already sharded, which is the case for input streams produced by add_input_zset.

Comment thread crates/dbsp/src/operator/output.rs Outdated
/// rebuild it, and give the relation a second time to readers that
/// already have it.
#[track_caller]
pub fn output_accumulated_stream_following_with_gid<B>(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is an overkill. This new mechanism makes sure that setting the distributed attribute doesn't lead to bootstrapping. I don't think it's an important case to cover and it's not worth the added complexity. I suggest we remove this mechanism and accept that (un)setting the distributed attribute can trigger bootstrapping.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done: removed in 3fdef33. Turning distributed on now starts a bootstrap, which sends the whole view to the connector again.

@ryzhyk

ryzhyk commented Nov 7, 2026

Copy link
Copy Markdown
Contributor

By examining individual features in this PR, claude found several serious issue. I will summarize them here, once comment per issue.

Issue1: Any change that changes cost estimation causes bootstrapping. In addition, upgrading to the new runtime with this PR will cause bootstrapping, even though docs say it shouldn't; however I don't think that's an issue.

Pipelines::new recomputes output_assignment from scratch on every start, and nothing records the previous assignment. So any change to the computed placement between two runs from the same checkpoint does the same damage.

What can move streams: anything that feeds assign_output_streams, which sorts streams by cost and breaks ties by name.
- Adding or removing a plain output connector on any stream, or turning distributed on or off for one. Distributed connectors don't count toward cost.
- Adding, removing or renaming a table or view, which changes the set of streams and their name order.
- A query change that makes the coordinator estimate a view as small (aggregated) instead of large, or the reverse.
- A coordinator release that changes the cost model or the tie-breaking. This is the upgrade case.

@ryzhyk

ryzhyk commented Nov 7, 2026

Copy link
Copy Markdown
Contributor

Issue 2. This one I haven't tried to understand in detail, and just passing on as is

A distributed Delta part can silently drop its snapshot or a transaction's changes (6; verified)
- Why: parts commit only in steps_processed, which runs only for queue entries that carry a step count. But DeltaTableWriter::batch_start clears pending_actions on every batch.
- Effect: a snapshot queued during an open transaction carries no count, so the commit's own change batch wipes the snapshot's files, or the reverse. No error is raised, and snapshot_sent is still checkpointed, so the snapshot is never resent.
- Fix: keep the current batch's files and the current step's files in separate lists.

I asked for a step-by-step scenario:

Setup

- Pipeline: multihost, 2 hosts, dev_tweaks.view_integrals: all_hosts, which is the default for a new pipeline. So each host holds part of every view.
- View: materialized view v = SELECT * FROM t. It already holds 300 committed rows, about 150 on each host.
- Output sink: delta_table_output with mode: append, update_mode: cdc, distributed: true, send_snapshot: true, paused: true.
- Leader: host 1, the host assigned to v. Host 0's part is a follower.

Sequence

Each step shows host 0; host 1 does the same with its own share.

1. While sink is paused: each step, CircuitThread::push_output gives it an empty entry. The snapshot stays pending until the connector starts.
2. POST /start_transaction:
   - From now on, CircuitThread::processed_records returns None after every step, because a transaction is open.
   - update_snapshot doesn't run during a transaction, so the newest stored snapshot is still rows 0..300.
3. The client inserts rows 300..310 into t inside the transaction.
4. POST /output_endpoints/sink/start: the coordinator relays it to both hosts.
5. The next step, still inside the transaction:
   - paused_outputs no longer contains sink, and its snapshot is pending.
   - Because the endpoint is distributed, push_output passes the step's own count along, which is None.
   - enqueue_latest_snapshot takes host 0's part of the stored snapshot (about 150 rows) and calls mark_snapshot_delivered.
   - It queues {snapshot, 150 rows, no step count}.
6. Host 0's output thread:
   - push_batch_to_encoder calls DeltaTableWriter::batch_start, which finds nothing pending.
   - encode writes Parquet files A for the 150 rows, so pending_actions = [A].
   - batch_end returns early, because a part keeps its files for the leader.
   - The entry has no step count, so steps_processed isn't called. Files A sit in the table directory, uncommitted.
7. POST /commit_transaction: at the step where the commit completes, processed_records becomes Some(C). The transaction's change stream is published; host 0's share is, say, 5 of the 10 new rows.
8. push_output: the snapshot is no longer pending, so the normal path queues {changes, 5 rows, step C}.
9. Host 0's output thread:
   - batch_start runs pending_actions.clear(), and files A are dropped.
   - encode writes files D0, so pending_actions = [D0], and batch_end returns early.
   - steps_processed(C) takes [D0], and OutputPart::commit reports only D0 to the leader.
10. Host 1, the leader:
    - Its own snapshot files B were dropped the same way when its change batch arrived.
    - CommitLeader::gather(C) receives D0, adds the leader's own D1, checks the paths, and commits D0 and D1 in one Delta commit.

Result

- The table holds only rows 300..310. The 300 snapshot rows never reach it.
- Files A and B remain in the table directory as orphans; VACUUM eventually removes them.
- No error is raised, and the connector stays healthy.
- The loss is permanent. Step 5 set the delivered flag. The next checkpoint stores snapshot_sent, so after a restart ControllerInner::add_output_endpoint treats the snapshot as already sent. Only changing the connector's definition clears the flag.

When it doesn't happen

- Host with no new rows in the commit: if a host has no new rows of its own, its change entry carries no data. batch_start never runs, and steps_processed(C) commits its snapshot files. With few new rows, only some hosts lose their share, so the table ends up with part of the snapshot.
- No transaction open at the start: the snapshot carries the step's count and is committed at once. I haven't traced whether a commit that spans several steps opens the same window without an API transaction.
- Plain Delta connectors: not affected, because their batch_end commits each batch immediately.

Test to reproduce it: two in-process hosts with this connector. Ingest 300 rows, start a transaction, insert 10 more, start the connector, step, then commit. Expect 310 rows in the table; I expect it to hold about 10.

Possible second bug (unverified): a start that lands on the step where a host publishes its share of a multi-step commit may make the snapshot skip that share. The pending-snapshot branch drops the step's changes, but the stored snapshot predates the transaction. If real, this would also affect plain connectors on main.

Next: waiting for your command.

@ryzhyk

ryzhyk commented Nov 7, 2026

Copy link
Copy Markdown
Contributor

Issue 3. A distributed input plus a distributed Delta output to a new table can never start

H3. A distributed input plus a distributed Delta output to a new table can never start (5–6; verified)
- Why: when a distributed Delta Lake or Iceberg input needs host 0 to choose a value first, the coordinator activates host 0 alone. Host 0 publishes its choices only after all its connectors, outputs included, have initialized.
- Effect: if the output's leader is on another host, host 0's part waits for a table that only the leader creates. It gives up after about 10 attempts, and every restart repeats this.
- Fix: publish choices once the inputs have initialized, or let the other parts open the table at their first commit.

@ryzhyk

ryzhyk commented Nov 7, 2026

Copy link
Copy Markdown
Contributor

Issue 4. Multihost snapshots are encoded one piece at a time (3; verified)

H4. Multihost snapshots are encoded one piece at a time (3; verified)
- Why: encode_snapshot_parts calls encode once per local worker part and once per ~1 MiB chunk from another host.
- Effect on Delta Lake:
  - Each call writes its own files, so thousands of small files (inferred).
  - With update_mode: merge, each call is its own commit, so readers see the snapshot half-applied (traced).
  - An encode error clears all earlier parts' files while the snapshot carries on, so the commit holds only the later parts.
- Fix: encode the local parts as one reader (whole), merge remote chunks into larger readers, and make Delta's error path drop only the failing call's files.

@ryzhyk

ryzhyk commented Nov 7, 2026

Copy link
Copy Markdown
Contributor

Medium importance issues (some looks pretty high to me):

Medium

- M5. Leases expire mid-snapshot (verified)
  - Why: all leases are taken up front, a lease's idle time resets only after a scan, hosts are read in order after the local parts, and the idle limit is 60 minutes.
  - Effect: if the earlier parts take over an hour to drain into the sink, the later hosts fail. This also happens under one_host, where those hosts' parts are empty, so it is a regression for large snapshots that used to be purely local.
- M6. A part's report larger than 2 MiB stalls the connector forever (verified)
  - Why: /coordination/output/commit uses actix's default 2 MiB JSON body limit, and PeerClient::report_output retries the resulting 413 error forever.
  - When: about 1–2k file entries per host in one step reaches the limit, for example the snapshot of a large view (size inferred).
- M7. send_snapshot on a distributed connector under one_host duplicates or loses rows (verified; both reviewers found it)
  - Why: only host H's part has a snapshot, taken at its own start step. The other parts send changes from their own start steps, and the coordinator starts each host independently.
  - Tables are not affected: a table's integral uses the same partition function (shard_batch) as the distributed changes. I discarded one reviewer's claim that tables are affected.
  - Fix: refuse this combination unless the layout is all_hosts, or have the leader pick one snapshot step for all parts.
- M8. A failed snapshot looks complete, or disappears
  - HTTP egress ends normally with partial data; the PR's own test asserts exactly this (verified).
  - A failed configured connector vanishes from /stats (traced).
  - A checkpoint taken mid-snapshot can store snapshot_sent for a snapshot that then fails, so it is never resent (traced).
- M9. A distributed output's pause state is never made consistent across hosts (traced)
  - The coordinator does this only for inputs.
  - After a restart some parts can stay paused and drop their share, while the merged status still says running.
- M10. Switching layouts under concurrent bootstrap serves empty views (runtime traced; I didn't check whether multihost pipelines use concurrent bootstrap)
  - Why: the switch marks no relation as affected.
  - Effect: ad hoc queries and new send_snapshot connectors see empty views until cutover.
- M11. HTTP/2 flow control shared between scans and leases (inferred)
  - When: over HTTPS without client certificates.
  - Effect: unread scan data can block a lease reply. The lease runs on the circuit thread and is retried every step, up to 30 s each time.

@ryzhyk

ryzhyk commented Nov 7, 2026

Copy link
Copy Markdown
Contributor
Low

- Version skew:
  - Old coordinator with a new runtime: host H's leader waits forever for reports that never come, because only the follower checks for host URLs (verified).
  - New coordinator with an old runtime: every host runs a full writer, and they race to create the table (traced).
- Background fetch task:
  - tokio::join!(fetch, forward) hangs when the reader goes away, which pins the other host's lease and data until restart.
  - An egress client that disconnects doesn't stop the fetch.
  - A closed channel counts as success, so a panic commits a partial snapshot.
- Leasing: leases are taken one host at a time on the circuit thread, up to 30 s × (hosts − 1) per step. A failed lease waits for the next step, which an idle pipeline may never take.
- Commit leader:
  - A part that skips a step deadlocks instead of getting an error.
  - CommitLeader::gather ignores close().
  - The 60 s "unhealthy" warning fires on large but healthy steps.
- Endpoint access:
  - The new host-to-host endpoints accept any caller the pipeline accepts, even on Solo.
  - The leader trusts report.host, so a forged report can complete a step.
  - Leases are unbounded.
  - Main's /coordination/* endpoints have the same exposure.
- Unused index names: the index names in the distributed set are never used. A distributed connector on view V gives V and all its indexes a distributed change stream.
- Clock skew: __feldera_ts now comes from several hosts' clocks, which can break the documented MERGE watermark when one host's clock lags.
- Chunk size: chunks have no upper bound for indexes with few distinct keys.
- Statistics: queued_batches drifts, and some merged counters mislead.

@blp
blp force-pushed the distributed-outputs branch from 02eb986 to 4fb7ade Compare November 11, 2026 22:51
@blp
blp changed the base branch from main to record-output-assignment November 11, 2026 22:52
@blp

blp commented Nov 11, 2026

Copy link
Copy Markdown
Member Author

This PR has changed since review; it will change again before it is ready for re-review.

@ryzhyk, the new bugfix PR #7942 that this is now based on is, however, ready for review now.

@blp
blp marked this pull request as draft November 11, 2026 23:00
@blp
blp force-pushed the distributed-outputs branch 2 times, most recently from 3fdef33 to b0f6c7b Compare November 12, 2026 00:56
@blp

blp commented Nov 13, 2026

Copy link
Copy Markdown
Member Author

@ryzhyk This PR is ready for re-review, together with feldera/cloud#2177. Here is where each issue from your review is now. Earlier replies give commit hashes, which later rebases changed, so this table names the commits by title.

Review item Fix Where
Issue 1: a change in cost estimation, or the upgrade, moves views and starts a bootstrap A checkpoint records the output assignment, and the coordinator keeps each stream on its checkpoint's host #7942 (approved, in the merge queue) and feldera/cloud#2210
Issue 2: a distributed Delta Lake part can drop its snapshot or a transaction's changes batch_start keeps a part's files until steps_processed hands them to the leader "Add distributed output connectors"
Issue 3: a distributed input plus a distributed Delta Lake output to a new table never starts A part that is not the leader opens the table at its first write, and a host publishes its input choices before its outputs start "Add distributed output connectors"
Issue 4: multihost snapshots are encoded one piece at a time Parts of about one file each, built with builders that spill to storage "Encode multihost snapshots in parts of about a file"; #7952 for one commit in merge mode
M5: leases expire during a snapshot Leases are touched every minute, and one_host takes no leases "Keep snapshot leases alive, and take none under one_host"
M6: a report larger than 2 MiB stalls the connector The limit is now 256 MiB, and a refused report is not retried "Add distributed output connectors"
M7: send_snapshot on a distributed connector under one_host Refused at startup "Add distributed output connectors"
M8: a failed snapshot looks complete, or a checkpoint records it as sent Egress ends with an error, a configured connector pauses and retries on start, and a snapshot counts as sent only when it is all written "Do not take a failed or unfinished snapshot for a sent one"
M9: the pause state of a distributed output differs between hosts The coordinator settles connector changes before a checkpoint feldera/cloud#2177, "Hold back connector pause changes during a checkpoint"; feldera/cloud#2217 to apply changes at a chosen step
M10: a layout switch under concurrent bootstrap serves empty views Refused, with advice to use concurrent_bootstrap=false "Refuse a concurrent bootstrap of a change of view layout"
M11: scans and leases share HTTP/2 flow control HTTP/1.1 only between hosts "Talk to the other hosts over HTTP/1.1 only"
Low items As in my reply to them The commits from "Stop a multihost snapshot's task" to "Count a multihost snapshot as one queued batch", the docs commit, #7957, #7958, and feldera/cloud#2212

Also, the mechanism that kept a view from being sent again when distributed was turned on is removed, as you suggested. Turning distributed on now starts a bootstrap.

Each new Rust test fails with its fix reverted. The PR description has the full list of commits and changes.

Merge order: #7942, then feldera/cloud#2210, then this PR, then feldera/cloud#2177. Each cloud PR first moves its submodule to the feldera main commit that it needs.

Base automatically changed from record-output-assignment to main November 13, 2026 00:55
Multihost pipelines gather output delta streams, and the integrals of
materialized views, to the host that the coordinator assigns to each
stream, but tables spread their integrals over all workers.  Add a table
to the documentation of Catalog that shows each case and explains why,
so that later changes to output placement can update it.

Signed-off-by: Ben Pfaff <blp@feldera.com>
///
/// After each warning interval of waiting, logs a warning and calls `on_wait` with how long it
/// has waited and the hosts that have reported.
pub fn gather(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bug (medium): leader and follower can deadlock when the follower is behind. gather handles a part that skipped ahead (it checks range(step + 1..)), but not one that is still waiting on an earlier step. If host 1 has a pending report of step 8 and the leader gathers step 10, then:

  • host 1 blocks in report(8) and never reports 10;
  • finish() would fail step 8, but it only runs after step 10 completes.

So neither side moves. Only shutdown ends it, and the connector turns unhealthy after 10 min. This can happen when the leader is recreated after disconnect_output, because output_part replaces the closed leader while a follower is still retrying an older step. A connector added through the API at different steps on different hosts can hit it too. Repro (fails; gather is still waiting after 3 s):

let leader = Arc::new(CommitLeader::new(2));
let rt = tokio::runtime::Runtime::new().unwrap();
let _behind = rt.spawn({ let l = leader.clone(); async move { l.report(1, 8, json!(1)).await } });
std::thread::sleep(Duration::from_millis(100));
let g = std::thread::spawn({ let l = leader.clone(); move || l.gather(10, &CancellationToken::new(), &mut |_, _| {}) });
std::thread::sleep(Duration::from_secs(3));
assert!(g.is_finished());

Suggested fix: at the start of gather(step), fail every pending step < step the same way finish does, so the waiting parts get an error and can report later steps.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 1c9b830, "[adapters] Fail a distributed commit report of a step that the leader passed": gather(step) now fails every pending report of an earlier step, as finish does, so the part gets an error and goes on to report the later step. Your repro is the new test a_part_that_is_behind_fails_instead_of_waiting, which also checks that the part's next report completes the gather. It fails without the fix.

Comment thread crates/connectors/src/server.rs Outdated
decoder.push(&response.body().await.unwrap());
let mut chunks = 0;
while let Some(chunk) = decoder.next_chunk() {
rows += unsafe { handle.deserialize_chunk(&chunk) }.len();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

missing (low, hard-rule gate): no // SAFETY: comment on this unsafe block. It's test code, but the rule covers every unsafe block. Copy the comment from static-connectors/src/catalog.rs, e.g. // SAFETY: this host's own scan of this relation's integral wrote chunk.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added, in "[adapters] Build multihost snapshots from the parts on every host", the commit that added this test.

@mythical-fred-oss

Copy link
Copy Markdown
Contributor

What I ran (debug builds, on the merge commit)

Suite Result
cargo test -p dbsp_connectors --lib (output_leader, snapshot_parts, distribution, part_file_tests, Delta follower/health) 34/34 pass
static-connectors --test connector_suites -- multihost 30/30 pass
connector_suites -- snapshot 47 pass, 4 fail: Postgres CDC tests, which need a Postgres server (environment, unrelated)
static-connectors --lib -- chunk (round trip of every SQL type through chunks) 3/3 pass

Gates: tests, docs and changelog, SQL type coverage for chunks, and the distributed/view_integrals flag are all in place. The unsafe impls have # Safety docs. One test unsafe block has no SAFETY comment (inline).

bug (medium): the commit leader deadlocks when a follower is still waiting on an earlier step than the one the leader gathers (see the inline comment and its repro). I'd fix this before merge, plus a unit test for it.

missing tests (medium):

  • A test where the leader is recreated (disconnect, then the connector is added again) while a follower is mid-report. This is the realistic trigger for the bug above.
  • A test where the peer stalls mid-scan without closing the connection. scan_once has no read or idle timeout, so only TCP keepalive (several minutes) or shutdown ends the snapshot.

@blp
blp force-pushed the distributed-outputs branch 2 times, most recently from 62eb86c to 209c7ae Compare November 13, 2026 05:48
@blp

blp commented Nov 13, 2026

Copy link
Copy Markdown
Member Author

Thanks for the QA pass. Pushed as 209c7ae:

Item Change
bug: leader and follower deadlock when the follower is behind Fixed in 1c9b830; see the inline reply
missing test: the peer stalls during a scan A real gap, fixed in 209c7ae, "[adapters] Retry a snapshot scan from a host that stops sending". A scan that gets no data for 5 minutes, for its response or between pieces of its body, fails, and fetch_part retries it from the chunks that arrived. New test stalled_scan_resumes_where_it_stopped, with a fake host that stops sending and keeps the connection open. It fails without the fix.
missing test: the leader is recreated while a part is in the middle of a report Not added. The unit test above covers the mechanism directly, since a recreated leader is a leader that gathers a later step than a pending report.
SAFETY comment Added

output_leader and snapshot_parts tests (24) and the multihost suite (30) pass, and clippy is clean.

@ryzhyk ryzhyk left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for addressing everything!

blp added 10 commits November 13, 2026 10:57
In a multihost pipeline, the integral of each materialized view was
gathered onto the host that the coordinator assigned to its stream.
That host held the whole view, and its workers did all of the view's
work.  The integral was there only so that a `send_snapshot` connector
on that host could read the whole view.  Ad hoc queries do not need it:
they read each worker's part of an integral on that worker's host.

With this change, a new multihost pipeline shards the integral of each
materialized view and index over all workers of all hosts, so that a
large view's state and work spread over all of them.

No host then has the whole view, so a multihost pipeline no longer
supports `send_snapshot`.  It refuses an output connector that sets it,
and HTTP egress refuses `send_snapshot=true`, whatever the layout of the
integrals.  An ad hoc query reads the whole view instead.

A checkpoint now records where the view integrals are, and a pipeline
that restarts from a checkpoint keeps that layout, so that an upgrade
does not move its state.  A checkpoint from before this change keeps
them on one host.  The new `dev_tweaks.view_integrals` setting overrides
the choice: `one_host` keeps the old layout, in case the new one has a
bug, and `all_hosts` moves a pipeline to the new one.  The persistent id
of an integral includes its range of workers, so a move in either
direction finds no state and rebuilds the integrals as a bootstrap,
which by default waits for approval, and the pipeline diff says why.

With the integrals on all hosts, a view's state does not depend on its
host, so a change of output assignment moves no state.

The multihost tests check the layout that each checkpoint records, each
layout switch, and the refusal of `send_snapshot`.  They count the rows
of the view on each host, as an ad hoc query reads them.

Signed-off-by: Ben Pfaff <blp@feldera.com>
An output connector with `distributed: true` now runs on every host of a
multihost pipeline, and each host's part writes the output of that
host's workers.  The null connector and the Delta Lake connector, with
`mode: append` and `update_mode: cdc`, support it.  On other
transports, it has no effect, as before: the connector runs on one host,
and the pipeline logs a warning.

A distributed Delta Lake output still commits each transaction once.
Each part writes its own data files and reports them, for each committed
step, to the part on the stream's assigned host, which commits every
part's files in one Delta commit and then answers the parts.  Only that
part creates the table.  It commits only files inside the table.  The
other parts wait for the table to exist.  Output buffering is refused
for now.

The parts call the leader over HTTPS, at the base URL of each host,
which the coordinator now sends in `CoordinationActivate::host_urls`.
When the pipeline does not require client certificates, each host trusts
the root CA of its own pipeline certificate chain, as the coordinator
does.  Without host URLs, as from an older coordinator, a part that is
not the leader refuses to start.

A part's report lists every file that it wrote in a step, so the leader
takes reports of up to 256 MiB, instead of actix's default 2 MiB limit
for JSON.  A part retries a report while the leader is busy, but not one
that the leader refuses, for example as too large, because the leader
would refuse it again.

A part of a distributed Delta Lake output that is not the leader waits,
as it initializes, for the leader to create the table.  When an input
connector needs host 0's choice, the coordinator activates host 0 alone
and activates the other hosts only after host 0 publishes its choices.
Host 0 published them only after all of its connectors initialized, so
with the leader on another host, host 0's part waited for a table that
nothing could create, and the pipeline could never start.  Two changes
fix this:

- A part that is not the leader no longer waits for the table while it
  initializes.  It opens the table before it writes its first files, in
  a step, and the pipeline runs its first step only after the
  coordinator has activated every host.  The connector traits now state
  this rule: a connector must not wait for another host while it
  initializes.

- A host of a multihost pipeline initializes its input connectors,
  publishes their choices, and only then initializes its output
  connectors, so that an output connector that breaks the rule cannot
  hold back the other hosts.

The in-process multihost test harness can run 3 hosts, passes each host
the other hosts' URLs, and can activate one host at a time, as the
coordinator does.

The connector's status combines the statuses of its parts.  The
distributed output of a relation is an output operator of its own, so a
restart that turns `distributed` on for a connector bootstraps that
operator, which sends the whole view to the connector again.

The docs describe distributed output connectors, and the Python platform
tests check a distributed Delta Lake output's deltas and resume.

Signed-off-by: Ben Pfaff <blp@feldera.com>
A restart that changes where a multihost pipeline keeps the state of its
materialized views (`dev_tweaks.view_integrals`) rebuilds the state of
every view.  A concurrent bootstrap leaves the views that it rebuilds
out of the live circuit until cutover, so every view would read as empty
to ad hoc queries until the rebuild ended.  Such a restart now fails, as
other changes that cannot be bootstrapped concurrently do, with advice
to restart with `concurrent_bootstrap=false`.

Signed-off-by: Ben Pfaff <blp@feldera.com>
…ut host URLs.

A coordinator that predates `CoordinationActivate::host_urls` sends no
host URLs.  The parts of a distributed Delta Lake output that are not
the leader refused to start without them, but the leader started, and
then waited forever for the reports of parts that could not send them.
Now every part refuses to start, the leader too.

Signed-off-by: Ben Pfaff <blp@feldera.com>
…ealthy.

The leader of a distributed Delta Lake output marked the connector
unhealthy after it had waited a minute for the other parts to report a
step.  The parts report a step only once they have written all of it,
so a large step on a healthy connector made it unhealthy.  Now the
connector stays healthy, with a description of the wait, until it has
waited 10 minutes.  The log still warns every minute.

Also, `CommitLeader::gather` did not notice `close()`, so a leader whose
connector stopped kept waiting.  Now it stops.

Signed-off-by: Ben Pfaff <blp@feldera.com>
The leader of a distributed Delta Lake output waits for every other part
to report a step.  A part that reported a later step without that one
had skipped it, and it waited for its later step, which the leader would
reach only after the step that it waited for, so the two waited for each
other forever.  Now the leader fails the step, and names the hosts that
skipped it, so that the connector reports an error.

Signed-off-by: Ben Pfaff <blp@feldera.com>
…need it.

A distributed output connector puts the stream that it reads, and its
index if it reads one, into the set of streams that get a change stream
sharded over all workers.  But the catalog checked an index by its
view's name, because it registers an index with its view's schema.  So
the index names in the set were never used, and a distributed connector
on a view gave the view and every index of the view a distributed change
stream, each with an exchange of its own.  A connector that read an
index got its stream only because its view's name was in the set.

Now the catalog checks an index by its own name, a view by its name and
by the name of the index that it doubles as, and the set holds a
connector's index if it reads one, and its stream otherwise.

Signed-off-by: Ben Pfaff <blp@feldera.com>
Each host writes `__feldera_ts` from its own clock, so the filter on its
maximum in the `MERGE INTO` example can skip rows if the hosts' clocks
differ.  And a distributed output needs a coordinator and pipeline hosts
from the same release.

Signed-off-by: Ben Pfaff <blp@feldera.com>
… passed.

The commit leader of a distributed output failed a part that had
skipped ahead of the step that it gathered, but not a part that was
behind it.  If a part waited for the answer to its report of step 8
while the leader gathered step 10, the part never reported step 10, and
the leader never finished step 10, which was the only place that failed
step 8.  Both waited until shutdown, and the connector became unhealthy
after 10 minutes.  This can happen when the leader is recreated, after
its connector is disconnected, while a part retries a report, or when
hosts add the connector at different steps.

Now gather fails every pending report of an earlier step, as finish
already did, so that the part gets an error and reports the later step.

Signed-off-by: Ben Pfaff <blp@feldera.com>
A report holds the Delta Lake `Add` action of each file that a part
wrote in a step, with the file's column statistics, not only a list of
files.  Each entry is about 1-4 KB, so one large step can exceed actix's
2 MiB JSON limit.

Signed-off-by: Ben Pfaff <blp@feldera.com>
@blp
blp force-pushed the distributed-outputs branch from 209c7ae to 577ea84 Compare November 13, 2026 19:10
@blp

blp commented Nov 13, 2026

Copy link
Copy Markdown
Member Author

Force-pushed as 577ea84: a multihost pipeline no longer supports send_snapshot, so this PR no longer builds snapshots from the parts on every host.

Change Effect
Removed: snapshot leases, /coordination/snapshot/*, the batch format of /coordination/adhoc/scan, serialize_chunks and deserialize_chunk, encoding in parts, batch_failed, and the HTTP/1.1 change About 4,300 fewer added lines. Review items issue 4, M5, M7, M8, and M11 no longer apply.
New: a multihost pipeline refuses send_snapshot, on a configured connector and on HTTP egress In "Spread materialized view integrals over all hosts". Test: multihost_send_snapshot_is_refused. The changelog and connector docs say so.
PeerClient moved to controller/peer_client.rs and keeps only the report to the leader Host URLs and the HTTPS trust now come in "Add distributed output connectors", their only user.
Tests that read a view through snapshot egress after a restart or a layout switch They now count the rows that each host holds, as an ad hoc query reads them. The layout test also checks where the rows are.

The commits after "Add distributed output connectors" are as before. #7957 is closed, because there are no snapshot chunks.

blp added 2 commits November 13, 2026 15:26
…ite.

When a part of a distributed Delta Lake output failed to write a batch,
`encode` dropped the part's files of the step and returned an error.
`steps_processed` then reported the files that the part still had,
often none, and the leader committed every other part's files.  A
reader saw part of the transaction, with no error on the other hosts.
This broke the promise in delta.md that a reader sees all of a
transaction's output or none of it, as with one host, where a failed
batch drops the whole step.

Now a part that fails to write in `encode` reports the failure, in the
new `error` field of `OutputPartReport`, instead of its files.  If any
part's report is a failure, the leader's own included, the leader
commits none of the step, and every part gets an error that names the
part that failed.  The files of the step are orphans that `VACUUM`
collects.

This covers a part that cannot open the leader's table, too.  Such a
part does not stop, as a comment said: it tries again for each batch.
Now each of its steps fails on every part, instead of committing
without its share.

The coordinator and the pipeline must be from the same release, as
delta.md already says, so the new field needs no compatibility.

New tests: `a_failed_part_fails_the_step`, for the leader, and
`a_failed_part_commits_none_of_the_step`, for a Delta Lake part that
fails one batch of a step.  Each fails with its half of the fix
reverted.

Signed-off-by: Ben Pfaff <blp@feldera.com>
…uts found.

- A restart warned that each output stream that the coordinator moved
  from its recorded host has its state rebuilt.  With view integrals on
  all hosts, the new default, no host has a stream's state, so the
  warning was false, and it could make an operator delay a safe
  restart.  Now it warns only when the integrals are on one host.

- A part that could not open its leader's table reported the error as a
  fatal transport error, although it tries again for its next batch, and
  `encode` returned the same error, which the output thread reported a
  second time.  Now only the output thread reports it, once.

- A part reached its leader by indexing the host URLs that the
  coordinator sent, so a coordinator that sent too few would make the
  output thread panic.  Now the controller refuses to start unless there
  is one URL for each host, and a report to a host with no URL is an
  error.

New test: `check_host_urls`, which fails without the check.

Signed-off-by: Ben Pfaff <blp@feldera.com>
@ryzhyk
ryzhyk added this pull request to the merge queue Nov 14, 2026
Merged via the queue into main with commit 9192193 Nov 14, 2026
1 check passed
@ryzhyk
ryzhyk deleted the distributed-outputs branch November 14, 2026 03:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

connectors Issues related to the adapters/connectors crate documentation Improvements or additions to documentation enterprise Issue related to Feldera Enterprise features. multihost Related to multihost or distributed pipelines performance python Pull requests that update python code rust Pull requests that update Rust code User-facing For PRs that lead to Feldera-user visible changes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants