Repository navigation
docs: Statistics after a resume, and the runtime status diagram - #7882
Conversation
| Standby -> Initializing [label=" /activate"]; | ||
| Initializing -> AwaitingApproval [label=" needs\n approval"]; | ||
| Initializing -> Replaying [label=" journal"]; | ||
| Initializing -> Bootstrapping [label=" changed"]; |
There was a problem hiding this comment.
bug (medium): The diagram has no Initializing -> ConcurrentBootstrapping edge. The lifecycle.md notes say changed plus concurrent_bootstrap=true on /start goes to ConcurrentBootstrapping, and modifying.md says the same. As drawn, the only way into ConcurrentBootstrapping is through AwaitingApproval.
| Initializing -> Bootstrapping [label=" changed"]; | |
| Initializing -> Bootstrapping [label=" changed"]; | |
| Initializing -> ConcurrentBootstrapping [label=" changed\n (concurrent)"]; |
(Regenerate the PNG after this.)
| Paused -> Running [label=" /resume"]; | ||
| Running -> Paused [label=" /pause"]; | ||
| Paused -> Suspended [label=" /stop"]; | ||
| Running -> Suspended [label=" /stop"]; |
There was a problem hiding this comment.
pitfall (low-medium): Only /stop?force=false writes a checkpoint and leads to Suspended. A forced /stop does not, so a reader may take the plain /stop label the wrong way. Suggest labelling both edges /stop\n force=false.
| the pipeline. They accumulate across checkpoint and resume. | ||
| the pipeline. They accumulate across checkpoint and resume, except | ||
| `records_processed_bytes_total`. See [Statistics after a | ||
| resume](/pipelines/fault-tolerance#statistics-after-a-resume). |
There was a problem hiding this comment.
bug (medium): records_processed_bytes_total is not the only exception here. ControllerStatus::new (controller/stats.rs ~L361) sets total_input_bytes: AtomicU64::new(0) on resume, so records_input_bytes_total also starts from zero. records_late_total comes from engine.late_records, which does not appear to be checkpointed either. Please list every exception here, and in the "Other statistics" paragraph in fault-tolerance.md too. Also apply the change to the generated metrics.md.
| |---------|----------------| | ||
| | Records processed by the pipeline | `records_processed_total`, `total_processed_records` in `/stats` | | ||
| | Records, bytes and errors for each input connector | `input_connector_*_total` metrics, input endpoint statistics in `/stats` | | ||
| | Records, bytes and errors for each output connector | `output_connector_*_total` metrics, output endpoint statistics in `/stats` | |
There was a problem hiding this comment.
bug (medium): output_connector_*_total claims too much. Only transmitted_records/bytes and the encode/transport error counts are in CheckpointOutputEndpointMetrics. Connector-specific counters such as output_connector_delta_merge_*_total (in-process atomics in delta_table/merge/metrics.rs) and the Kafka ones start from zero. The same goes for the "These metrics accumulate across checkpoint and resume" sentences in the Input/Output Connectors sections of metrics.md, which also cover gauges, histograms and the *_delta_*/*_iceberg_* counters. Suggest naming the restored metrics explicitly: output_connector_records_total, output_connector_bytes_total, output_connector_errors_*_total, and the matching input metrics.
|
Review summary (docs-only PR)
Once these are fixed it looks good. The commit message is fine. |
657276f to
fc2c658
Compare
|
Thanks. All four points are fixed, and the PR is now two commits, one for each issue (force-pushed).
-- Claude Code, replying at @blp's request. |
| @@ -0,0 +1,39 @@ | |||
| // Source for runtime_status.png. Regenerate with: | |||
| // dot -Tpng -Gdpi=200 runtime_status.dot -o runtime_status.png | |||
| digraph runtime_status { | |||
There was a problem hiding this comment.
Should the diagram also show stopping/editing/restarting a pipeline?
There was a problem hiding this comment.
Stopping, editing and restarting change the resources status, not the runtime status: the runtime status exists only while the resources status is Provisioned. So I left them out of this diagram, and added a note that says this and links to the resources status diagram. A restart is the "Pipeline process starts" entry at the top.
-- Claude Code, replying at @blp's request.
| ## Statistics after a resume | ||
|
|
||
| A checkpoint also stores the pipeline's record counters. When a | ||
| pipeline resumes from a checkpoint, these counters go back to their |
There was a problem hiding this comment.
"go back" is confusing. "Continue" is better.
There was a problem hiding this comment.
Changed to "continue from their values at the time of the checkpoint".
-- Claude Code, replying at @blp's request.
| | Records, bytes and errors for each input connector | `input_connector_records_total`, `input_connector_bytes_total`, `input_connector_errors_transport_total`, `input_connector_errors_parse_total` | the same counters in each input endpoint's metrics | | ||
| | Records, bytes and errors for each output connector | `output_connector_records_total`, `output_connector_bytes_total`, `output_connector_errors_transport_total`, `output_connector_errors_encode_total` | the same counters in each output endpoint's metrics | | ||
|
|
||
| Work that the pipeline did after the checkpoint is not counted. The |
There was a problem hiding this comment.
Work performed between a checkpoint and a crash is not counted as performed when the pipeline resumes.
There was a problem hiding this comment.
Used your sentence.
-- Claude Code, replying at @blp's request.
| total. It writes a checkpoint after 500,000 records, reads 250,000 | ||
| more, and then fails. After it resumes, it reports 500,000 processed | ||
| records. When it has read all of its input, it reports 1,000,000. | ||
| Thus, the counters show each record one time, not the 1,250,000 |
There was a problem hiding this comment.
The counter shows 1M - committed work, not 1,2M - performed work.
There was a problem hiding this comment.
Changed to "the 1,000,000 records of committed work, not the 1,250,000 records of work that the pipeline actually performed".
-- Claude Code, replying at @blp's request.
| resume, so a destination can receive more records than the counters | ||
| show. | ||
|
|
||
| Because a resume moves these counters back, Prometheus sees a counter |
There was a problem hiding this comment.
can cause the counters to decrease
There was a problem hiding this comment.
Changed to "Because a resume can cause these counters to decrease".
-- Claude Code, replying at @blp's request.
| Because a resume moves these counters back, Prometheus sees a counter | ||
| reset, in the same way as when a process restarts. | ||
|
|
||
| Other statistics are for the current pipeline process only and start |
There was a problem hiding this comment.
Changed to "for the current incarnation of the pipeline only, that is, for the current pipeline process".
-- Claude Code, replying at @blp's request.
|
I will approve, but this can be improved. |
Thanks, your wording is better. I'll adopt it. |
A checkpoint stores the pipeline's record counters and the record, byte and error totals of each connector. On resume, these go back to their values at the checkpoint, and other statistics start from zero. The fault tolerance page now has a section about this, and the metrics reference says which of its metrics accumulate across a resume. Fixes #6057. Signed-off-by: Ben Pfaff <blp@feldera.com>
The lifecycle page said that the diagram would be added in the future. It now has the diagram, notes on the transitions, and the ConcurrentBootstrapping and Synchronizing statuses that the table did not list. runtime_status.dot is the source for the diagram. Fixes #4810. Signed-off-by: Ben Pfaff <blp@feldera.com>
5a13051 to
62e27fc
Compare
Two docs issues, one commit each.
Statistics after a resume (#6057)
New section in
pipelines/fault-tolerance.md. A resume moves these counters back to their checkpoint values:records_input_total,records_processed_total, and the records, bytes and error totals of each input and output connector. Work done after the checkpoint is counted again when the pipeline does it again, so each input record is counted once. Output sent again after a resume, with at-least-once fault tolerance or with a connector that is not fault tolerant, is not in the counters. Everything else starts from zero:records_input_bytes_total,records_processed_bytes_total,records_late_total, connector-specific counters (Delta Lake, Iceberg, Kafka), gauges, histograms and process metrics. The checkpointed fields are incrates/connectors/src/controller/checkpoint.rsandControllerStatus::newincontroller/stats.rs.The metrics reference (
metrics.md.inand the generatedmetrics.md) now names the metrics that accumulate across a resume in each section and links to the new section.Runtime status diagram (#4810)
pipelines/lifecycle.mdhad "The diagram will be added in the future." It now has the diagram, notes on each kind of transition, including/startwithconcurrent_bootstrap=true(start, restart,Unavailable, errors, suspend, multihostCoordination), and table rows forConcurrentBootstrappingandSynchronizing, which were missing. The transitions come fromget_statusincrates/connectors/src/server.rs.The diagram is made with Graphviz from
runtime_status.dot, which is committed next to the PNG. The other status diagrams are hand-drawn in the brand style, so a designer may want to redraw this one to match.Fixes #6057.
Fixes #4810.