Repository navigation
[dbsp] Keep the scheduler's broadcasts when it is prepared again - #7884
Conversation
Preparing a scheduler again gave it new broadcasts, and so new exchanges. At the cutover of a concurrent bootstrap, each host prepares the main circuit's scheduler again on its own, so a host that cut over first could step the circuit and send on a new exchange before another host created it. The receiving host then panicked with `UnknownExchange`, which fails the pipeline since commit d5fa739 ("[dbsp] Report panics in exchange receive tasks"). Now the scheduler keeps its broadcasts when it is prepared again. This is safe because every worker prepares again between the same two steps. Co-authored-by: Ben Pfaff <blp@feldera.com> Signed-off-by: Leonid Ryzhyk <ryzhyk@gmail.com>
|
LGTM. Keeping the two
|
|
Suggestion for one more test: check the new rule in The driver test in Possible shape:
I reverted only the |
The concurrent bootstrap tests catch a scheduler that creates exchanges when it is prepared again only if one host cuts over late enough for the other hosts to send first. The new test checks the rule in `Scheduler::prepare` without depending on timing: preparing the scheduler again must take no IDs from `Runtime::sequence_next`, from which every exchange takes its ID. Signed-off-by: Leonid Ryzhyk <ryzhyk@gmail.com>
Preparing a scheduler again gave it new broadcasts, and so new exchanges. At the cutover of a concurrent bootstrap, each host prepares the main circuit's scheduler again on its own, so a host that cut over first could step the circuit and send on a new exchange before another host created it. The receiving host then panicked with
UnknownExchange, which fails the pipeline since commit d5fa739 ("[dbsp] Report panics in exchange receive tasks").Now the scheduler keeps its broadcasts when it is prepared again. This is safe because every worker prepares again between the same two steps.
Linked Issue
Subsumes #7877
Describe Manual Test Plan
Checklist
Breaking Changes?
Mark if you think the answer is yes for any of these components:
Describe Incompatible Changes