← 输入框|github.com|刷新
Skip to content

[dbsp] Name the storage settings in the storage I/O failure message - #7911

Merged
Karakatiza666 merged 1 commit into
mainfrom
storage-failure-hint
Nov 10, 2026
Merged

Karakatiza666 merged 1 commit into
mainfrom
storage-failure-hint

Conversation

@Karakatiza666

@Karakatiza666 Karakatiza666 commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

The storage I/O failure panic (crates/dbsp/src/trace/ord/file.rs) told the user to use "a larger, initially empty storage volume", but did not say how. It now names the settings and links to the docs:

 * Reconfigure the container to use a larger storage volume by increasing
   `resources.storage_max` (combine it with `resources.storage_min` for
   autoscaling: then `min` is the reserved baseline, and `max` the autoscaling
   limit); see https://docs.feldera.com/pipelines/configuration#resizing-a-running-pipeline.
   If this requires clearing storage, and the pipeline had a checkpoint
   synchronized to object storage (such as S3), consider setting
   `storage.backend.config.sync.start_from_checkpoint` in the runtime
   configuration to restart from the checkpoint.

"Initially empty" is removed: on Kubernetes, PATCH deployment_config grows the volume in place and keeps the state. Storage must be cleared only to change these fields in the runtime configuration, so the checkpoint hint now applies only to that case. The linked section covers both ways.

Refs #6092 (requested in the issue comments). This PR does not fix the banner cut-off in the web console.

Comment thread crates/dbsp/src/trace/ord/file.rs Outdated
Comment on lines +54 to +55
* Reconfigure the container to use a larger, initially empty storage volume
by increasing `resources.storage_max` in the runtime configuration.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pitfall (medium): storage_max alone is not always the setting that controls the volume size. docs/pipelines/configuration.mdx says: "Storage grows to storage_min, else storage_max."

  • If only storage_min is set, raising storage_max does not make the volume bigger. It turns on autoscaling instead, and the API refuses that when enableStorageAutoscaling is off.
  • The runtime config field can only be edited once storage has been cleared. "Initially empty" implies this, but most users won't read it that way.
  • On Kubernetes, a running or stopped pipeline can also grow in place with PATCH deployment_config and keep its state. That option seems more useful here than clearing storage.

Possible wording: "…by clearing storage and then increasing resources.storage_min or resources.storage_max (whichever is set) in the runtime configuration." Maybe also add a bullet that points to resizing a running pipeline.

@mythical-fred-oss

Copy link
Copy Markdown
Contributor

What I ran

Check Result
cargo check --locked -p dbsp pass
pre-commit run --files crates/dbsp/src/trace/ord/file.rs pass

Gates: This changes only the text of a panic message, so not having a test is fine. No unsafe code, dependency, or API changes. The commit message is good.

pitfall (medium): The new advice is accurate only when storage_min is unset and storage has already been cleared. See the inline comment for the storage_min and in-place PATCH deployment_config cases.

pitfall (low/medium): This panic also fires on the open-source and Docker runners, which don't size a volume from resources.storage_max. There the advice does nothing. Something like "(Kubernetes / Enterprise)" would prevent confusion.

@Karakatiza666 Karakatiza666 changed the title [dbsp] Name the storage setting in the storage I/O failure message [dbsp] Name the storage settings in the storage I/O failure message Nov 10, 2026
@Karakatiza666
Karakatiza666 marked this pull request as ready for review November 10, 2026 13:56
Comment thread crates/dbsp/src/trace/ord/file.rs Outdated
Comment on lines +54 to +56
* Reconfigure the container to use a larger storage volume by increasing
`resources.storage_min` for a higher baseline, or `resources.storage_max`
for a higher autoscaling ceiling; see

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pitfall (medium): "storage_max for a higher autoscaling ceiling" is only true when storage_min is also set. If only storage_max is set (the common case without autoscaling), storage_max is the size of the volume (config.rs: "Without storage_min, this much storage is reserved up front"). A user in that setup could read this as "storage_max doesn't apply to me" and raise storage_min instead, which turns on autoscaling, needs storage cleared, and may be blocked by enableStorageAutoscaling. Also, if neither field is set (the runner's default size), a resize can't be done at all. Maybe something like:

Suggested change
* Reconfigure the container to use a larger storage volume by increasing
`resources.storage_min` for a higher baseline, or `resources.storage_max`
for a higher autoscaling ceiling; see
* Reconfigure the container to use a larger storage volume by increasing
`resources.storage_max` (or, with storage autoscaling, `resources.storage_min`
and/or `resources.storage_max`); see

@mythical-fred-oss

Copy link
Copy Markdown
Contributor

Summary: This PR only changes the wording of the panic message, so the test and docs gates don't apply. Nothing tests for the old text, the doc anchor #resizing-a-running-pipeline exists in pipelines/configuration.mdx, and the commit message is fine.

Ran Result
cargo check --locked -p dbsp ✅
pre-commit run --files crates/dbsp/src/trace/ord/file.rs ✅
  • pitfall (medium): the storage_min/storage_max advice is wrong when autoscaling is off (see inline). Otherwise LGTM.

The panic message now names `resources.storage_min` and
`resources.storage_max` and links to the docs on resizing storage. It no
longer says the larger volume must be empty: a resize keeps the state, so
the hint about restarting from a synchronized checkpoint now applies only
when storage is cleared.

Refs #6092
@Karakatiza666
Karakatiza666 added this pull request to the merge queue Nov 10, 2026
Merged via the queue into main with commit e5be949 Nov 10, 2026
1 check passed
@Karakatiza666
Karakatiza666 deleted the storage-failure-hint branch November 10, 2026 17:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants