← 输入框|github.com|刷新
Skip to content

[dbsp] Keep the scheduler's broadcasts when it is prepared again - #7884

Merged
ryzhyk merged 2 commits into
mainfrom
scheduler-keeps-exchanges
Nov 7, 2026
Merged

ryzhyk merged 2 commits into
mainfrom
scheduler-keeps-exchanges

Conversation

@ryzhyk

@ryzhyk ryzhyk commented Nov 6, 2026

Copy link
Copy Markdown
Contributor

Preparing a scheduler again gave it new broadcasts, and so new exchanges. At the cutover of a concurrent bootstrap, each host prepares the main circuit's scheduler again on its own, so a host that cut over first could step the circuit and send on a new exchange before another host created it. The receiving host then panicked with UnknownExchange, which fails the pipeline since commit d5fa739 ("[dbsp] Report panics in exchange receive tasks").

Now the scheduler keeps its broadcasts when it is prepared again. This is safe because every worker prepares again between the same two steps.

Linked Issue

Subsumes #7877

Describe Manual Test Plan

Checklist

  • Unit tests added/updated
  • Integration tests added/updated
  • Documentation updated
  • Changelog updated

Breaking Changes?

Mark if you think the answer is yes for any of these components:

Describe Incompatible Changes

Preparing a scheduler again gave it new broadcasts, and so new exchanges.
At the cutover of a concurrent bootstrap, each host prepares the main
circuit's scheduler again on its own, so a host that cut over first could
step the circuit and send on a new exchange before another host created
it.  The receiving host then panicked with `UnknownExchange`, which fails
the pipeline since commit d5fa739 ("[dbsp] Report panics in exchange
receive tasks").

Now the scheduler keeps its broadcasts when it is prepared again.  This
is safe because every worker prepares again between the same two steps.

Co-authored-by: Ben Pfaff <blp@feldera.com>
Signed-off-by: Leonid Ryzhyk <ryzhyk@gmail.com>
@ryzhyk
ryzhyk requested a review from blp November 6, 2026 22:53
@ryzhyk ryzhyk added the DBSP core Related to the core DBSP library label Nov 6, 2026
@mythical-fred-oss

Copy link
Copy Markdown
Contributor

LGTM. Keeping the two Broadcasts across re-prepare is the right fix. DynamicScheduler is the only Scheduler impl, and Balancer::prepare creates no exchanges, so the new no-new-exchanges contract on Scheduler::prepare holds.

What I ran Result
cargo test -p dbsp --lib replay_tests::test_concurrent (52 tests) pass, 13s
The same with only the scheduler fix reverted (new test kept), 3 runs fails every run with UnknownExchange(N), so the regression test catches the bug
cargo test -p dbsp --lib circuit::, 3 runs 155/155 on one run; on the other two, one of test_panic_in_tokio_merger_runtime / test_unicast_command_after_background_panic failed
pre-commit run --files <changed> (clippy, fmt, doc) pass
  • observation (unrelated to this PR): both of those panic tests also fail locally with this PR's scheduler change reverted, and they run on 1 worker (Broadcast::SingleThreaded). They look like existing timing-dependent tests.
  • pitfall (low): the test's sleep(100ms) only makes the race more likely to show up. With the fix in place it can't make the test fail, so it doesn't break the flaky-test gate.
  • missing coverage (low): only the multi-host cutover path is skewed. The abort path (bootstrap.rs, "Reactivate the full circuit") also calls prepare again on each host but isn't exercised with skewed hosts.

@blp

blp commented Nov 6, 2026

Copy link
Copy Markdown
Member

Suggestion for one more test: check the new rule in Scheduler::prepare (schedule.rs, "Invoking it again must not create exchanges") directly.

The driver test in replay_tests.rs finds the race only when host 0's 100 ms delay lets the other hosts send first. A direct test does not depend on timing. It would also find a future change that makes prepare (or something it calls, such as balancer().prepare) create an exchange.

Possible shape:

  1. Build a circuit with more than one worker. With one worker, Broadcast::new returns SingleThreaded and takes no exchange ID, so the test would always pass.
  2. In a worker, read Runtime::sequence_next(), prepare the scheduler again (for example with Some(active) and then with None, as a bootstrap does), and read Runtime::sequence_next() again.
  3. Expect the second value to be the first value plus one, because each read uses one ID.

I reverted only the dynamic_scheduler.rs change and kept the new test driver: 43 of the 52 test_concurrent* tests fail with UnknownExchange. With the PR as written, all 52 pass. This result could also go in the empty "Manual Test Plan" section.

@blp blp left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like this better than #7877.

@ryzhyk
ryzhyk added this pull request to the merge queue Nov 7, 2026
@ryzhyk
ryzhyk removed this pull request from the merge queue due to a manual request Nov 7, 2026
The concurrent bootstrap tests catch a scheduler that creates exchanges
when it is prepared again only if one host cuts over late enough for the
other hosts to send first.  The new test checks the rule in
`Scheduler::prepare` without depending on timing: preparing the scheduler
again must take no IDs from `Runtime::sequence_next`, from which every
exchange takes its ID.

Signed-off-by: Leonid Ryzhyk <ryzhyk@gmail.com>
@ryzhyk
ryzhyk added this pull request to the merge queue Nov 7, 2026
Merged via the queue into main with commit 941e19d Nov 7, 2026
1 check passed
@ryzhyk
ryzhyk deleted the scheduler-keeps-exchanges branch November 7, 2026 09:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

DBSP core Related to the core DBSP library

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants