← 输入框|github.com|刷新
Skip to content

docs: Statistics after a resume, and the runtime status diagram - #7882

Merged
blp merged 2 commits into
mainfrom
docs-checkpoint-lifecycle
Nov 10, 2026
Merged

blp merged 2 commits into
mainfrom
docs-checkpoint-lifecycle

Conversation

@blp

@blp blp commented Nov 6, 2026 •

Copy link
Copy Markdown
Member

Two docs issues, one commit each.

Statistics after a resume (#6057)

New section in pipelines/fault-tolerance.md. A resume moves these counters back to their checkpoint values: records_input_total, records_processed_total, and the records, bytes and error totals of each input and output connector. Work done after the checkpoint is counted again when the pipeline does it again, so each input record is counted once. Output sent again after a resume, with at-least-once fault tolerance or with a connector that is not fault tolerant, is not in the counters. Everything else starts from zero: records_input_bytes_total, records_processed_bytes_total, records_late_total, connector-specific counters (Delta Lake, Iceberg, Kafka), gauges, histograms and process metrics. The checkpointed fields are in crates/connectors/src/controller/checkpoint.rs and ControllerStatus::new in controller/stats.rs.

The metrics reference (metrics.md.in and the generated metrics.md) now names the metrics that accumulate across a resume in each section and links to the new section.

Runtime status diagram (#4810)

pipelines/lifecycle.md had "The diagram will be added in the future." It now has the diagram, notes on each kind of transition, including /start with concurrent_bootstrap=true (start, restart, Unavailable, errors, suspend, multihost Coordination), and table rows for ConcurrentBootstrapping and Synchronizing, which were missing. The transitions come from get_status in crates/connectors/src/server.rs.

The diagram is made with Graphviz from runtime_status.dot, which is committed next to the PNG. The other status diagrams are hand-drawn in the brand style, so a designer may want to redraw this one to match.

Fixes #6057.
Fixes #4810.

@blp blp added documentation Improvements or additions to documentation ft Fault tolerant, distributed, and scale-out implementation labels Nov 6, 2026
@blp blp self-assigned this Nov 6, 2026
Standby -> Initializing [label=" /activate"];
Initializing -> AwaitingApproval [label=" needs\n approval"];
Initializing -> Replaying [label=" journal"];
Initializing -> Bootstrapping [label=" changed"];

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bug (medium): The diagram has no Initializing -> ConcurrentBootstrapping edge. The lifecycle.md notes say changed plus concurrent_bootstrap=true on /start goes to ConcurrentBootstrapping, and modifying.md says the same. As drawn, the only way into ConcurrentBootstrapping is through AwaitingApproval.

Suggested change
Initializing -> Bootstrapping [label=" changed"];
Initializing -> Bootstrapping [label=" changed"];
Initializing -> ConcurrentBootstrapping [label=" changed\n (concurrent)"];

(Regenerate the PNG after this.)

Paused -> Running [label=" /resume"];
Running -> Paused [label=" /pause"];
Paused -> Suspended [label=" /stop"];
Running -> Suspended [label=" /stop"];

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pitfall (low-medium): Only /stop?force=false writes a checkpoint and leads to Suspended. A forced /stop does not, so a reader may take the plain /stop label the wrong way. Suggest labelling both edges /stop\n force=false.

the pipeline. They accumulate across checkpoint and resume.
the pipeline. They accumulate across checkpoint and resume, except
`records_processed_bytes_total`. See [Statistics after a
resume](/pipelines/fault-tolerance#statistics-after-a-resume).

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bug (medium): records_processed_bytes_total is not the only exception here. ControllerStatus::new (controller/stats.rs ~L361) sets total_input_bytes: AtomicU64::new(0) on resume, so records_input_bytes_total also starts from zero. records_late_total comes from engine.late_records, which does not appear to be checkpointed either. Please list every exception here, and in the "Other statistics" paragraph in fault-tolerance.md too. Also apply the change to the generated metrics.md.

|---------|----------------|
| Records processed by the pipeline | `records_processed_total`, `total_processed_records` in `/stats` |
| Records, bytes and errors for each input connector | `input_connector_*_total` metrics, input endpoint statistics in `/stats` |
| Records, bytes and errors for each output connector | `output_connector_*_total` metrics, output endpoint statistics in `/stats` |

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bug (medium): output_connector_*_total claims too much. Only transmitted_records/bytes and the encode/transport error counts are in CheckpointOutputEndpointMetrics. Connector-specific counters such as output_connector_delta_merge_*_total (in-process atomics in delta_table/merge/metrics.rs) and the Kafka ones start from zero. The same goes for the "These metrics accumulate across checkpoint and resume" sentences in the Input/Output Connectors sections of metrics.md, which also cover gauges, histograms and the *_delta_*/*_iceberg_* counters. Suggest naming the restored metrics explicitly: output_connector_records_total, output_connector_bytes_total, output_connector_errors_*_total, and the matching input metrics.

@mythical-fred-oss

Copy link
Copy Markdown
Contributor

Review summary (docs-only PR)

Check Result
pre-commit run --files <changed> passed
Statements checked against server.rs::get_status and controller/checkpoint.rs / stats.rs mostly right, see inline comments
Links (#statistics-after-a-resume, #process-metrics, #concurrent-bootstrapping) headings exist
Gates (tests, unsafe, deps, CI, breaking changes) not applicable, no code changes
  • bug (medium): the diagram is missing the Initializing → ConcurrentBootstrapping edge (/start with concurrent_bootstrap=true). The notes describe this edge, but the PNG does not show it.
  • bug (medium): the list of counters that reset on resume is incomplete. records_input_bytes_total also starts from zero, and probably records_late_total too. output_connector_*_total claims too much, because connector-specific counters like the Delta merge and Kafka ones are not checkpointed.

Once these are fixed it looks good. The commit message is fine.

@blp
blp force-pushed the docs-checkpoint-lifecycle branch from 657276f to fc2c658 Compare November 6, 2026 19:48
@blp

blp commented Nov 6, 2026

Copy link
Copy Markdown
Member Author

Thanks. All four points are fixed, and the PR is now two commits, one for each issue (force-pushed).

  • Diagram: added Initializing -> ConcurrentBootstrapping (changed (concurrent)), labelled both edges into Suspended as /stop force=false, and regenerated the PNG.
  • Counters that start from zero: you are right. ControllerStatus::new restores only the record counts, and records_late_total is not checkpointed. The fault tolerance page now lists records_input_bytes_total, records_processed_bytes_total and records_late_total as starting from zero.
  • Connector metrics: the table now names exactly the restored metrics: {input,output}_connector_records_total, _bytes_total, and the two _errors_*_total of each. Connector-specific counters, gauges and histograms are listed as starting from zero.
  • metrics.md: each section now names the metrics that accumulate across a resume. I made the same change in metrics.md.in and the generated metrics.md.

-- Claude Code, replying at @blp's request.

@blp
blp requested a review from mihaibudiu November 6, 2026 19:50
@@ -0,0 +1,39 @@
// Source for runtime_status.png. Regenerate with:
// dot -Tpng -Gdpi=200 runtime_status.dot -o runtime_status.png
digraph runtime_status {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should the diagram also show stopping/editing/restarting a pipeline?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stopping, editing and restarting change the resources status, not the runtime status: the runtime status exists only while the resources status is Provisioned. So I left them out of this diagram, and added a note that says this and links to the resources status diagram. A restart is the "Pipeline process starts" entry at the top.

-- Claude Code, replying at @blp's request.

## Statistics after a resume

A checkpoint also stores the pipeline's record counters. When a
pipeline resumes from a checkpoint, these counters go back to their

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"go back" is confusing. "Continue" is better.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changed to "continue from their values at the time of the checkpoint".

-- Claude Code, replying at @blp's request.

| Records, bytes and errors for each input connector | `input_connector_records_total`, `input_connector_bytes_total`, `input_connector_errors_transport_total`, `input_connector_errors_parse_total` | the same counters in each input endpoint's metrics |
| Records, bytes and errors for each output connector | `output_connector_records_total`, `output_connector_bytes_total`, `output_connector_errors_transport_total`, `output_connector_errors_encode_total` | the same counters in each output endpoint's metrics |

Work that the pipeline did after the checkpoint is not counted. The

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Work performed between a checkpoint and a crash is not counted as performed when the pipeline resumes.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Used your sentence.

-- Claude Code, replying at @blp's request.

total. It writes a checkpoint after 500,000 records, reads 250,000
more, and then fails. After it resumes, it reports 500,000 processed
records. When it has read all of its input, it reports 1,000,000.
Thus, the counters show each record one time, not the 1,250,000

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The counter shows 1M - committed work, not 1,2M - performed work.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changed to "the 1,000,000 records of committed work, not the 1,250,000 records of work that the pipeline actually performed".

-- Claude Code, replying at @blp's request.

resume, so a destination can receive more records than the counters
show.

Because a resume moves these counters back, Prometheus sees a counter

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can cause the counters to decrease

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changed to "Because a resume can cause these counters to decrease".

-- Claude Code, replying at @blp's request.

Because a resume moves these counters back, Prometheus sees a counter
reset, in the same way as when a process restarts.

Other statistics are for the current pipeline process only and start

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

current incarnation?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changed to "for the current incarnation of the pipeline only, that is, for the current pipeline process".

-- Claude Code, replying at @blp's request.

@mihaibudiu

Copy link
Copy Markdown
Contributor

I will approve, but this can be improved.

@blp

blp commented Nov 6, 2026

Copy link
Copy Markdown
Member Author

I will approve, but this can be improved.

Thanks, your wording is better. I'll adopt it.

blp added 2 commits November 6, 2026 12:00
A checkpoint stores the pipeline's record counters and the record,
byte and error totals of each connector.  On resume, these go back to
their values at the checkpoint, and other statistics start from zero.
The fault tolerance page now has a section about this, and the metrics
reference says which of its metrics accumulate across a resume.

Fixes #6057.

Signed-off-by: Ben Pfaff <blp@feldera.com>
The lifecycle page said that the diagram would be added in the future.
It now has the diagram, notes on the transitions, and the
ConcurrentBootstrapping and Synchronizing statuses that the table did
not list.  runtime_status.dot is the source for the diagram.

Fixes #4810.

Signed-off-by: Ben Pfaff <blp@feldera.com>
@blp
blp force-pushed the docs-checkpoint-lifecycle branch from 5a13051 to 62e27fc Compare November 6, 2026 20:00
@blp
blp added this pull request to the merge queue Nov 9, 2026
Merged via the queue into main with commit baff480 Nov 10, 2026
1 check passed
@blp
blp deleted the docs-checkpoint-lifecycle branch November 10, 2026 01:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation ft Fault tolerant, distributed, and scale-out implementation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Ensure that documentation is clear about semantics of checkpoints Runtime status diagram in documentation

2 participants