← 输入框|github.com|刷新
Skip to content

[dbsp] Keep the circuit running after a failed checkpoint - #7846

Merged
blp merged 1 commit into
mainfrom
checkpoint-error-keeps-circuit
Nov 6, 2026
Merged

blp merged 1 commit into
mainfrom
checkpoint-error-keeps-circuit

Conversation

@blp

@blp blp commented Nov 4, 2026

Copy link
Copy Markdown
Member

A failed checkpoint no longer kills the circuit. PrepareCheckpoint only takes in-memory snapshots of operator state, so the circuit stays consistent after it fails. The controller already treats a checkpoint error as non-fatal and retries later.

Closes #7844.

Describe Manual Test Plan

The new test test_failed_checkpoint_keeps_circuit fails both with the old kill-on-error (Runtime(Terminated)) and with an early return that does not drain the other workers' replies (Expected CommitProgress response, got CheckpointPrepared([])).

Checklist

  • Unit tests added/updated
  • Integration tests added/updated
  • Documentation updated
  • Changelog updated

@blp blp added bug Something isn't working DBSP core Related to the core DBSP library labels Nov 4, 2026
@blp blp self-assigned this Nov 4, 2026
@mythical-fred-oss

Copy link
Copy Markdown
Contributor

LGTM.

What I ran

Check Result
cargo test --locked -p dbsp --lib test_failed_checkpoint_keeps_circuit, 20 runs pass every time, no flakiness
cargo test --locked -p dbsp --lib circuit::dbsp_handle 32 pass, 1 fail: test_panic_in_tokio_merger_runtime. It also fails with the PR reverted, so the PR did not cause it (it times out in this sandbox).

Checked the core claim. PrepareCheckpoint is side-effect free. CircuitHandle::checkpoint only calls Node::checkpoint. I read the z1, transaction_z1, trace, accumulate_trace, window, output and spine save implementations. Each one only snapshots or serializes in memory, and none does an exchange. A failure part-way through therefore leaves the workers consistent. Draining the remaining replies before returning is the right fix for the stale-reply problem.

Test gaps (medium):

  • The new non-fatal Ok(Err(e)) => return Err(e) arm in unicast_command has no test. Nothing unicasts PrepareCheckpoint today, so this arm is unreachable for now.
  • No test covers every worker failing at once. The current test fails only worker 0.
  • No controller-level test checks that a failed periodic checkpoint keeps the pipeline running and that a later checkpoint succeeds. This is the behaviour A worker error on a DBSP command, such as a failed checkpoint, kills the circuit #7844 reports.

@blp
blp added this pull request to the merge queue Nov 4, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Nov 5, 2026
@mihaibudiu
mihaibudiu added this pull request to the merge queue Nov 5, 2026
@mythical-fred-oss

Copy link
Copy Markdown
Contributor

Verdict: infrastructure, not this PR. High confidence.

  • Failing job: Integration Tests / Runtime Integration Tests (arm64)
  • The only failure is tests/runtime/test_udf.py::TestUDF::test_legacy_variant_udf, which gets 401 credential rejected (invalid or expired token / API key) on its very first API call (log lines 455, 607-608, 654, 748-749) — the shared CI OIDC token going stale mid-run, not a test or product bug.
  • This PR's diff is entirely crates/dbsp/src/circuit/dbsp_handle.rs (checkpoint/circuit resume logic) and has nothing to do with the Python client, UDFs, or CI auth.
  • This is a known, recurring issue: tracked in #7048, where it's hit several unrelated PRs over the past weeks. I've added this run's evidence there as another occurrence.

Next step: re-queueing should be fine — the same shared instance instability could recur, but there's nothing to fix in this PR.

@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Nov 5, 2026
@blp
blp added this pull request to the merge queue Nov 5, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Nov 5, 2026
A worker error reply to any command made `DBSPHandle` kill the runtime,
so a failed checkpoint stopped the pipeline.  `PrepareCheckpoint` only
takes in-memory snapshots of operator state, so a failure leaves the
circuit consistent.  Each command now says whether an error from it is
fatal, and only `PrepareCheckpoint` is not.

After a non-fatal error, `broadcast_command` still reads the replies of
the other workers, so that no stale reply is left for the next command.

The regression test for #7075 used a failed checkpoint to make the
circuit thread kill the runtime.  A failed checkpoint no longer does
that, so this adds a test-only command, `InjectFatalError`, behind the
new `testing` feature of `dbsp`, that every worker fails fatally.
`Controller::inject_fatal_error` sends it from the circuit thread.  The
test, now `worker_error_does_not_deadlock`, checks that a failed
checkpoint leaves the circuit running and that a fatal error does not
deadlock.

Fixes #7844.

Signed-off-by: Ben Pfaff <blp@feldera.com>
@blp
blp force-pushed the checkpoint-error-keeps-circuit branch from c0e5749 to 3c6ce0a Compare November 5, 2026 21:05
@blp
blp enabled auto-merge November 5, 2026 21:05
@blp
blp disabled auto-merge November 5, 2026 21:45
@blp
blp added this pull request to the merge queue Nov 5, 2026
Merged via the queue into main with commit 855cd31 Nov 6, 2026
1 check passed
@blp
blp deleted the checkpoint-error-keeps-circuit branch November 6, 2026 01:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working DBSP core Related to the core DBSP library

Projects

None yet

Development

Successfully merging this pull request may close these issues.

A worker error on a DBSP command, such as a failed checkpoint, kills the circuit

2 participants