Repository navigation
Keep only the newest checkpoint on storage - #7905
Conversation
| /// Deletes the checkpoints that are older than `uuid`, except those in | ||
| /// `keep`. A host of a multihost pipeline does this when the coordinator | ||
| /// says that every host has `uuid`. | ||
| fn gc_checkpoints_before( |
There was a problem hiding this comment.
is this using the timestamp encoded in the uuid?
There was a problem hiding this comment.
No. "Older" means written earlier. The checkpointer keeps checkpoint_list in the order the checkpoints were written: each committed checkpoint goes onto the end (crates/dbsp/src/circuit/checkpointer.rs:437). gc_checkpoints_before finds uuid in that list by position and keeps it and everything after it (checkpointer.rs:580-581). The UUID's timestamp is never read.
If uuid is not in the list, nothing is deleted and a warning is logged, so a stale or wrong UUID cannot delete anything.
-- Claude Code, replying at @blp's request.
79363f2 to
fac88f8
Compare
|
I asked claude to check if this PR can somehow cause data loss. It didn't find new bugs, but it uncovered two existing ones, one of which got worse with this PR. Reporting them here, although they may require separate PRs. |
fac88f8 to
3d13bb7
Compare
A pipeline kept its two newest checkpoints. The older one is not needed: the newest checkpoint lists every batch that the pipeline used when it wrote it, so the older checkpoint's other files are unused. Keeping it could use as much storage again as the pipeline's state. `gc_checkpoint` now takes the number of checkpoints to keep. A single host keeps 1. A host of a multihost pipeline still keeps 2, because a restart resumes from the newest checkpoint that every host has. Fixes #7035. Signed-off-by: Ben Pfaff <blp@feldera.com>
… has the newest. A host of a multihost pipeline keeps its previous checkpoint, because a restart resumes from the newest checkpoint that every host has. After a checkpoint succeeds on every host, the coordinator can now say so with `POST /coordination/checkpoint/complete`, which names this host's copy of that checkpoint. The host then deletes its older checkpoints, with the new `gc_checkpoints_before`. The request names the checkpoint, instead of only saying "delete the older one", so that a late request cannot delete a checkpoint that another host still lacks. Signed-off-by: Ben Pfaff <blp@feldera.com>
3d13bb7 to
63b58ef
Compare
A pipeline kept its two newest checkpoints on storage. The older one is not needed on a single host: the newest checkpoint lists every batch that the pipeline used when it wrote it, so the older checkpoint's other files are unused. With a long delay between checkpoints, the older checkpoint could use as much storage again as the pipeline's state.
Commits
Keep only the newest checkpoint on a single host (Fixes [dbsp] Keeping two checkpoints around is wasteful #7035).
Checkpointer::gc_checkpoint(except, multihost: bool)decides how many checkpoints to keep: 1 for a single host, 2 for a host of a multihost pipeline. The count is now private, ingc_keeping, instead of the oldMIN_CHECKPOINT_THRESHOLD.DBSPHandleand theCircuitDriverengine trait takemultihosttoo, and the controller passes it after each checkpoint.Delete a multihost host's older checkpoint once every host has the newest.
POST /coordination/checkpoint/complete {"uuid": ...}and newgc_checkpoints_before(except, uuid), which keepsuuidand every newer checkpoint.Testing
gc_keep_one_then_restore(dbsp)gc_commits(dbsp, extended)single_host_keeps_one_checkpoint(controller)multihost_keeps_two_checkpoints_until_complete(controller)Also ran the dbsp checkpoint and GC tests, the connector-suite checkpoint, suspend, and restore tests, and the Kafka FT and Postgres CDC tests against local Redpanda and Postgres.
test_pipeline_lifecycle.pyno longer expects exactly 2 checkpoints.