Repository navigation
[pipeline-manager] Grow a running pipeline's storage with PATCH - #7656
Conversation
| { | ||
| return Err(format!("storage_min ({min}) exceeds storage_max ({max})")); | ||
| } | ||
| if let (Some(old), Some(new)) = (storage_size(¤t), storage_size(&new)) |
There was a problem hiding this comment.
pitfall (medium): The shrink check compares only the configured sizes. In two cases it lets through a size the volume can't shrink to:
- No size configured. If
storage_minandstorage_maxare both unset,storage_size(¤t)isNone, so any value is accepted. The test at L493 confirms this:storage_max: "10M"is accepted even though the runner's default PVC is far larger. The bad value is saved and kept across restarts. - Autoscaling. Once the PVC has grown past
storage_min(say min=1G, max=8G, PVC now 6G), a patch ofstorage_min: "2G"orstorage_max: "2G"passes because it is compared with 1G, not 6G. The ceiling ends up below the volume's real size.
Could we either refuse a storage patch when no size is configured, or compare against the size the runner last reported? If neither, the docs should say the runner may reject such a patch, and what happens then (log only? applied at next start?).
|
What I ran (debug build, merge commit)
#[test]
fn storage_patch_on_old_spelling_and_mixed_aliases() {
let mut old = deployment();
old["resources"].as_object_mut().unwrap().remove("storage_max");
old["resources"]["storage_mb_max"] = json!(1000);
let e = try_patch(&old, &resize(json!({"storage_max": "500M"}))).unwrap_err();
assert!(e.contains("cannot shrink"), "{e}");
let ok = try_patch(&old, &resize(json!({"storage_max": "2G"}))).unwrap();
assert_eq!(ok["resources"]["storage_max"], json!("2G"));
assert!(ok["resources"].get("storage_mb_max").is_none());
assert!(parse_deployment_config_patch(
&resize(json!({"storage_min": "2G", "storage_mb_min": 1000}))).is_err());
}The gates for tests, docs, changelog, feature flag ( |
| && new < old | ||
| { | ||
| return Err(format!( | ||
| "storage cannot shrink from {old} to {new} (the size is storage_min, else storage_max)" |
There was a problem hiding this comment.
pitfall (medium): With autoscaling on, this compares against the declared storage_min, not the volume's actual size. The autoscaler may already have grown the PVC past storage_min. Example: min=1G, max=8G, PVC autoscaled to 6G. A patch of storage_min: "4G" or storage_max: "2G" returns 200 here (the test at L467 even expects storage_max: 2G to pass), yet neither can take effect: a PVC can't shrink, and the ceiling sits below the current size. Either check against the reported storage size from the status details, or document that the runner treats these as no-ops, so a user doesn't read a 200 as "applied".
| if cpu_memory_edited { | ||
| merge_cpu_memory(&mut stored.global.resources, &new.resources); | ||
| } | ||
| if storage_edited { |
There was a problem hiding this comment.
tech-debt (medium): I don't think production can reach this branch. update_pipeline (operations/pipeline.rs ~L637-656) refuses storage_min/storage_max edits unless storage is Cleared. With storage cleared, the automaton's start (pipeline_automata.rs ~L1242) ignores the previous deployment_config. So "a storage edit keeps a CPU resize" never happens, and the same goes for most of what V38 guards against. cpu_memory_and_storage_edits_are_merged_separately tests a state that can't occur. The docs line "until … storage, are changed in the runtime configuration" has the same problem: storage can only be changed after a clear, and the clear already resets the resize. Consider dropping the storage half here and in the docs, or explain in a comment why you want it as a safeguard. Also, only_a_cpu_or_memory_edit_updates_the_deployment_config is a stale test name now.
|
Review: storage resize via
Missing (medium), test coverage: no test covers a deployment config stored before the size rename ( #[test]
fn storage_resize_reads_legacy_spelling_and_mixed_units() {
let mut legacy = deployment();
legacy["resources"].as_object_mut().unwrap().remove("storage_max");
legacy["resources"]["storage_mb_max"] = json!(1000);
let new = try_patch(&legacy, &resize(json!({"storage_max": "1Gi"}))).unwrap();
assert_eq!(new["resources"]["storage_max"], json!("1Gi"));
assert!(try_patch(&legacy, &resize(json!({"storage_max": "999M"}))).unwrap_err().contains("cannot shrink"));
let mut gi = deployment();
gi["resources"]["storage_max"] = json!("1Gi");
assert!(try_patch(&gi, &resize(json!({"storage_max": "1G"}))).unwrap_err().contains("shrink"));
assert!(try_patch(&gi, &resize(json!({"storage_max": "1Gi"}))).is_ok()); // same size is not a shrink
}Pitfall (medium), cross-repo: the manager only stores the new size. Nothing in this repo applies it: the automaton sends only |
7f2848b to
d834ed1
Compare
d834ed1 to
3532036
Compare
With --enable-pipeline-resize, a PATCH of deployment_config can set storage_min, storage_max, and volume_attributes_class while a single-host pipeline runs. A PATCH can only increase storage_min or storage_max, and only one that is set, so storage stays fixed or autoscaling, and the volume stays between them. The class can change but not be removed. A value outside the storage limits of the installation is refused. The changes last across restarts until storage is cleared. A runtime_config edit of CPU and memory, or of the class, replaces them. A class edited on a stopped pipeline before this upgrade is not synced, and the changelog says to edit it again. Unset fields are left out of a serialized DynamicConfig, so an older runtime still accepts a memory update.
3532036 to
aee1984
Compare
PATCH /v0/pipelines/{name}withdeployment_configcan now grow the storage of a running pipeline throughresources.storage_minandresources.storage_max, and change itsresources.volume_attributes_class(experimental,--enable-pipeline-resize).storage_min, elsestorage_max, the size new storage starts atstorage_minorstorage_max, and only one that is already set. So storage stays fixed or autoscaling, the volume stays between the two, and storage with the runner's default size cannot be resizedstorage_max; a patch can raise either boundruntime_configstorage_classstays a start-time setting: Kubernetes cannot change it on a bound PVC.runtime_configedit copies CPU and memory, or the volume attributes class, only when that group changed, so a CPU edit keeps a storage resize. Storage is not copied: it can only be edited with storage cleared, and such a start takes it fromruntime_config.DynamicConfig, so a pipeline on an older runtime still accepts the memory update the automaton sends.The Kubernetes runner side is feldera/cloud#2129. This
PATCHis checked against the storage limits that #7712 gave the API, with a 400InvalidDeploymentPatch.Testing: unit tests in
deployment.rs(including a storedstorage_mb_max, mixed spellings, andGi/Gunits),runtime_config_storage_requestsfor thePATCHlimits, and the model proptest. End to end on EKS 1.35 with EBS CSI: before the increase-only rule: storage grew 10 to 20 GiB online, the class changed live, shrink, mode change, and empty class were refused, both changes survived a restart, and a class outside the allowlist was refused by the runner.test_deployment_resourcesin the platform tests was not run.