← 输入框|github.com|刷新
Skip to content

[pipeline-manager] Grow a running pipeline's storage with PATCH - #7656

Merged
xiasongh merged 1 commit into
mainfrom
song/pipeline-storage-resize
Nov 4, 2026
Merged

xiasongh merged 1 commit into
mainfrom
song/pipeline-storage-resize

Conversation

@xiasongh

@xiasongh xiasongh commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

PATCH /v0/pipelines/{name} with deployment_config can now grow the storage of a running pipeline through resources.storage_min and resources.storage_max, and change its resources.volume_attributes_class (experimental, --enable-pipeline-resize).

Rule Behavior
Size Storage grows to storage_min, else storage_max, the size new storage starts at
Change A patch can only increase storage_min or storage_max, and only one that is already set. So storage stays fixed or autoscaling, the volume stays between the two, and storage with the runner's default size cannot be resized
Autoscaling (both set) Keeps expanding up to storage_max; a patch can raise either bound
Volume attributes class Can change, cannot be removed (Kubernetes does not allow it); an empty class is refused
Restart The change lasts until storage is cleared, or until CPU and memory, or the class, are edited in runtime_config

storage_class stays a start-time setting: Kubernetes cannot change it on a bound PVC.

  • A runtime_config edit copies CPU and memory, or the volume attributes class, only when that group changed, so a CPU edit keeps a storage resize. Storage is not copied: it can only be edited with storage cleared, and such a start takes it from runtime_config.
  • No migration for the class: a class edited on a stopped pipeline before the upgrade is not synced, and the changelog says to set it again.
  • Unset storage fields and class are left out of a serialized DynamicConfig, so a pipeline on an older runtime still accepts the memory update the automaton sends.
  • The web-console client is regenerated.

The Kubernetes runner side is feldera/cloud#2129. This PATCH is checked against the storage limits that #7712 gave the API, with a 400 InvalidDeploymentPatch.

Testing: unit tests in deployment.rs (including a stored storage_mb_max, mixed spellings, and Gi/G units), runtime_config_storage_requests for the PATCH limits, and the model proptest. End to end on EKS 1.35 with EBS CSI: before the increase-only rule: storage grew 10 to 20 GiB online, the class changed live, shrink, mode change, and empty class were refused, both changes survived a restart, and a class outside the allowlist was refused by the runner. test_deployment_resources in the platform tests was not run.

{
return Err(format!("storage_min ({min}) exceeds storage_max ({max})"));
}
if let (Some(old), Some(new)) = (storage_size(&current), storage_size(&new))

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pitfall (medium): The shrink check compares only the configured sizes. In two cases it lets through a size the volume can't shrink to:

  1. No size configured. If storage_min and storage_max are both unset, storage_size(&current) is None, so any value is accepted. The test at L493 confirms this: storage_max: "10M" is accepted even though the runner's default PVC is far larger. The bad value is saved and kept across restarts.
  2. Autoscaling. Once the PVC has grown past storage_min (say min=1G, max=8G, PVC now 6G), a patch of storage_min: "2G" or storage_max: "2G" passes because it is compared with 1G, not 6G. The ceiling ends up below the volume's real size.

Could we either refuse a storage patch when no size is configured, or compare against the size the runner last reported? If neither, the docs should say the runner may reject such a patch, and what happens then (log only? applied at next start?).

@mythical-fred-oss

Copy link
Copy Markdown
Contributor

What I ran (debug build, merge commit)

Check Result
cargo test -p feldera-types resource_config_tests ✅ 4/4
cargo test -p pipeline-manager --lib db::types::deployment ✅ 18/18, plus 1 QA test of mine (below)
db::test::stopped_deployment_* (V37 and V38 migrations) and db_impl_behaves_like_model proptest ✅ (I needed RUST_MIN_STACK to stop the debug-build stack overflowing; that predates this PR)
pre-commit run --files <changed> ❌ see below
test_deployment_resources platform test not run (needs a cluster with resize enabled)
  • missing (medium): the generated web-console client is stale. pre-commit (the "Update API client" hook) regenerates js-packages/web-console/src/lib/services/manager/{sdk,types}.gen.ts and adds storage_min/storage_max plus the new doc text. Please commit the regenerated files, or the pre-merge-queue check will leave a dirty tree. The Clippy hook failed here because postgresql_embedded could not build in my sandbox; I don't think that is caused by this PR.
  • missing (medium): no test where storage is configured but the runner is not involved. Nothing tests storage on a multihost pipeline, a patch that mixes storage_min and storage_mb_min, or a stored pre-rename storage_mb_max config. I checked the last two locally and they behave correctly. Suggested addition to deployment.rs's mod test:
#[test]
fn storage_patch_on_old_spelling_and_mixed_aliases() {
    let mut old = deployment();
    old["resources"].as_object_mut().unwrap().remove("storage_max");
    old["resources"]["storage_mb_max"] = json!(1000);
    let e = try_patch(&old, &resize(json!({"storage_max": "500M"}))).unwrap_err();
    assert!(e.contains("cannot shrink"), "{e}");
    let ok = try_patch(&old, &resize(json!({"storage_max": "2G"}))).unwrap();
    assert_eq!(ok["resources"]["storage_max"], json!("2G"));
    assert!(ok["resources"].get("storage_mb_max").is_none());
    assert!(parse_deployment_config_patch(
        &resize(json!({"storage_min": "2G", "storage_mb_min": 1000}))).is_err());
}

The gates for tests, docs, changelog, feature flag (--enable-pipeline-resize) and commit message are met. I found no new dependencies and no unsafe code.

@xiasongh
xiasongh marked this pull request as ready for review October 27, 2026 17:08
&& new < old
{
return Err(format!(
"storage cannot shrink from {old} to {new} (the size is storage_min, else storage_max)"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pitfall (medium): With autoscaling on, this compares against the declared storage_min, not the volume's actual size. The autoscaler may already have grown the PVC past storage_min. Example: min=1G, max=8G, PVC autoscaled to 6G. A patch of storage_min: "4G" or storage_max: "2G" returns 200 here (the test at L467 even expects storage_max: 2G to pass), yet neither can take effect: a PVC can't shrink, and the ceiling sits below the current size. Either check against the reported storage size from the status details, or document that the runner treats these as no-ops, so a user doesn't read a 200 as "applied".

if cpu_memory_edited {
merge_cpu_memory(&mut stored.global.resources, &new.resources);
}
if storage_edited {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

tech-debt (medium): I don't think production can reach this branch. update_pipeline (operations/pipeline.rs ~L637-656) refuses storage_min/storage_max edits unless storage is Cleared. With storage cleared, the automaton's start (pipeline_automata.rs ~L1242) ignores the previous deployment_config. So "a storage edit keeps a CPU resize" never happens, and the same goes for most of what V38 guards against. cpu_memory_and_storage_edits_are_merged_separately tests a state that can't occur. The docs line "until … storage, are changed in the runtime configuration" has the same problem: storage can only be changed after a clear, and the clear already resets the resize. Consider dropping the storage half here and in the docs, or explain in a comment why you want it as a safeguard. Also, only_a_cpu_or_memory_edit_updates_the_deployment_config is a stale test name now.

@mythical-fred-oss

Copy link
Copy Markdown
Contributor

Review: storage resize via PATCH deployment_config. No feldera-pr-checks gate fails: the feature is behind --enable-pipeline-resize, it has docs, unit tests and a migration test, and the commit message is good.

What I ran Result
cargo test -p feldera-types resource_config_tests ✅ 4 passed
cargo test -p pipeline-manager db::types::deployment ✅ 18 passed
db::test::stopped_deployment_* (V37/V38, embedded PG) ✅ 2 passed
db::test::db_impl_behaves_like_model (proptest, RUST_MIN_STACK=16M) ✅ passed
pre-commit run --files <changed> ✅ all hooks passed
python/tests/platform::test_deployment_resources ⚠️ not run here or by the author (needs a K8s cluster with resize enabled)

Missing (medium), test coverage: no test covers a deployment config stored before the size rename (storage_mb_max) and then patched, or mixed Gi/G units in the shrink check. I wrote and ran these locally and they pass; worth adding:

#[test]
fn storage_resize_reads_legacy_spelling_and_mixed_units() {
    let mut legacy = deployment();
    legacy["resources"].as_object_mut().unwrap().remove("storage_max");
    legacy["resources"]["storage_mb_max"] = json!(1000);
    let new = try_patch(&legacy, &resize(json!({"storage_max": "1Gi"}))).unwrap();
    assert_eq!(new["resources"]["storage_max"], json!("1Gi"));
    assert!(try_patch(&legacy, &resize(json!({"storage_max": "999M"}))).unwrap_err().contains("cannot shrink"));
    let mut gi = deployment();
    gi["resources"]["storage_max"] = json!("1Gi");
    assert!(try_patch(&gi, &resize(json!({"storage_max": "1G"}))).unwrap_err().contains("shrink"));
    assert!(try_patch(&gi, &resize(json!({"storage_max": "1Gi"}))).is_ok()); // same size is not a shrink
}

Pitfall (medium), cross-repo: the manager only stores the new size. Nothing in this repo applies it: the automaton sends only memory_max, and the runtime ignores storage. Until the runner PR lands, a 200 on a storage patch on a resize-enabled cluster changes nothing. Please merge the two PRs together, or say in the docs that the runner support is needed.

Comment thread crates/feldera-types/src/config.rs Resolved
Comment thread crates/feldera-types/src/config.rs Resolved
@xiasongh
xiasongh force-pushed the song/pipeline-storage-resize branch from d834ed1 to 3532036 Compare November 3, 2026 19:20
With --enable-pipeline-resize, a PATCH of deployment_config can set
storage_min, storage_max, and volume_attributes_class while a
single-host pipeline runs. A PATCH can only increase storage_min or
storage_max, and only one that is set, so storage stays fixed or
autoscaling, and the volume stays between them. The class can change
but not be removed. A value outside the storage limits of the
installation is refused.

The changes last across restarts until storage is cleared. A
runtime_config edit of CPU and memory, or of the class, replaces them.
A class edited on a stopped pipeline before this upgrade is not
synced, and the changelog says to edit it again.

Unset fields are left out of a serialized DynamicConfig, so an older
runtime still accepts a memory update.
@xiasongh
xiasongh force-pushed the song/pipeline-storage-resize branch from 3532036 to aee1984 Compare November 3, 2026 19:48
@xiasongh
xiasongh added this pull request to the merge queue Nov 3, 2026
Merged via the queue into main with commit 9344785 Nov 4, 2026
1 check passed
@xiasongh
xiasongh deleted the song/pipeline-storage-resize branch November 4, 2026 01:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants