Repository navigation
[dbsp] Name the storage settings in the storage I/O failure message - #7911
Conversation
| * Reconfigure the container to use a larger, initially empty storage volume | ||
| by increasing `resources.storage_max` in the runtime configuration. |
There was a problem hiding this comment.
pitfall (medium): storage_max alone is not always the setting that controls the volume size. docs/pipelines/configuration.mdx says: "Storage grows to storage_min, else storage_max."
- If only
storage_minis set, raisingstorage_maxdoes not make the volume bigger. It turns on autoscaling instead, and the API refuses that whenenableStorageAutoscalingis off. - The runtime config field can only be edited once storage has been cleared. "Initially empty" implies this, but most users won't read it that way.
- On Kubernetes, a running or stopped pipeline can also grow in place with
PATCH deployment_configand keep its state. That option seems more useful here than clearing storage.
Possible wording: "…by clearing storage and then increasing resources.storage_min or resources.storage_max (whichever is set) in the runtime configuration." Maybe also add a bullet that points to resizing a running pipeline.
|
What I ran
Gates: This changes only the text of a panic message, so not having a test is fine. No unsafe code, dependency, or API changes. The commit message is good. pitfall (medium): The new advice is accurate only when pitfall (low/medium): This panic also fires on the open-source and Docker runners, which don't size a volume from |
62d0d12 to
b6d32ba
Compare
| * Reconfigure the container to use a larger storage volume by increasing | ||
| `resources.storage_min` for a higher baseline, or `resources.storage_max` | ||
| for a higher autoscaling ceiling; see |
There was a problem hiding this comment.
pitfall (medium): "storage_max for a higher autoscaling ceiling" is only true when storage_min is also set. If only storage_max is set (the common case without autoscaling), storage_max is the size of the volume (config.rs: "Without storage_min, this much storage is reserved up front"). A user in that setup could read this as "storage_max doesn't apply to me" and raise storage_min instead, which turns on autoscaling, needs storage cleared, and may be blocked by enableStorageAutoscaling. Also, if neither field is set (the runner's default size), a resize can't be done at all. Maybe something like:
| * Reconfigure the container to use a larger storage volume by increasing | |
| `resources.storage_min` for a higher baseline, or `resources.storage_max` | |
| for a higher autoscaling ceiling; see | |
| * Reconfigure the container to use a larger storage volume by increasing | |
| `resources.storage_max` (or, with storage autoscaling, `resources.storage_min` | |
| and/or `resources.storage_max`); see |
|
Summary: This PR only changes the wording of the panic message, so the test and docs gates don't apply. Nothing tests for the old text, the doc anchor
|
The panic message now names `resources.storage_min` and `resources.storage_max` and links to the docs on resizing storage. It no longer says the larger volume must be empty: a resize keeps the state, so the hint about restarting from a synchronized checkpoint now applies only when storage is cleared. Refs #6092
b6d32ba to
c0c7ecf
Compare
The storage I/O failure panic (
crates/dbsp/src/trace/ord/file.rs) told the user to use "a larger, initially empty storage volume", but did not say how. It now names the settings and links to the docs:"Initially empty" is removed: on Kubernetes,
PATCH deployment_configgrows the volume in place and keeps the state. Storage must be cleared only to change these fields in the runtime configuration, so the checkpoint hint now applies only to that case. The linked section covers both ways.Refs #6092 (requested in the issue comments). This PR does not fix the banner cut-off in the web console.