← 输入框|github.com|刷新
Skip to content

[connectors] Step an idle circuit under memory pressure - #7909

Merged
xiasongh merged 1 commit into
mainfrom
song/idle-step-memory-pressure-70ff87
Nov 11, 2026
Merged

xiasongh merged 1 commit into
mainfrom
song/idle-step-memory-pressure-70ff87

Conversation

@xiasongh

@xiasongh xiasongh commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

The ad-hoc snapshot keeps the last step's batches in memory, even after a merge or a spill replaces them (SpineSnapshot, update_snapshot). An idle pipeline takes no step, so this memory stays in use. In a k3d test, anon memory stayed at 2.40 GB after a spill, and one step dropped it to 0.61 GB.

Under high memory pressure, the statistics thread requests a step after 5 s with no step, at most once each 5 s. Only when no step runs and no transaction is open. The step goes through the existing request_step() path.

Tests:

  • idle_step_timer: the delay, the rate limit, and the restart after a step.
  • idle_pipeline_steps_under_memory_pressure: with a 1 MB budget and no input, the pipeline steps.
  • idle_pipeline_does_not_step_without_memory_pressure: without pressure, it does not.

@xiasongh
xiasongh marked this pull request as ready for review November 10, 2026 04:31
Comment thread crates/connectors/src/controller.rs Resolved
@mythical-fred-oss

Copy link
Copy Markdown
Contributor

What I ran

Check Result
cargo test -p dbsp_connectors --lib idle_step_timer ✅
cargo test -p static-connectors --test connector_suites idle_pipeline_steps ✅ 6 out of 6 runs, about 5 s each
The same test with controller.request_step() commented out ❌ times out after 15 s, so the test does catch the bug
cargo fmt --check ✅

Gates: tests ✅. Flakiness: no flakiness seen. The wait has a 10 s margin over the 1 s stats tick, which looks safe. No connector or format change, no unsafe, no new dependencies.

Tests the PR doesn't cover (medium):

  • A single step that runs longer than IDLE_STEP_INTERVAL (see the inline comment).
  • Pressure below High (for example, an unlimited max_rss): the idle pipeline should take no step. Nothing currently checks that the gate is off.
  • Rate limit end to end: with constant pressure, roughly one step every 5 s, not a busy loop.

@mythical-fred-oss

Copy link
Copy Markdown
Contributor

missing (medium): negative test. I ran this locally: it passes on the PR and fails when the >= High gate is removed. One caveat: with no max_rss, the limit comes from the host's available memory, so a CI runner that is short on memory could push pressure to High.

diff --git a/crates/connectors/src/controller/test.rs b/crates/connectors/src/controller/test.rs
index 9dc538b..3949956 100644
--- a/crates/connectors/src/controller/test.rs
+++ b/crates/connectors/src/controller/test.rs
@@ -11325,3 +11325,35 @@ pub fn paused_connector_holds_transaction(engine: &dyn PipelineTestEngine) {
     assert!(paused().is_empty());
     controller.stop().unwrap();
 }
+
+/// Without memory pressure, an idle pipeline takes no step.
+#[engine_test]
+pub fn idle_pipeline_does_not_step_without_memory_pressure(engine: &dyn PipelineTestEngine) {
+    init_test_logger();
+
+    let config: PipelineConfig = serde_json::from_value(json!({
+        "name": "test",
+        "workers": 1,
+        "clock_resolution_usecs": null,
+        "inputs": {}
+    }))
+    .unwrap();
+    let controller = engine
+        .controller(
+            &identity(&[None]),
+            &config,
+            Box::new(|e, _| panic!("error: {e}")),
+        )
+        .unwrap();
+    controller.start();
+
+    let steps = || controller.status().global_metrics.total_initiated_steps();
+    let before = steps();
+    let timeout = IDLE_STEP_INTERVAL + Duration::from_secs(3);
+    assert!(
+        wait(|| steps() > before, timeout.as_millis() as u64).is_err(),
+        "an idle pipeline without memory pressure stepped"
+    );
+
+    controller.stop().unwrap();
+}

Comment thread crates/connectors/src/controller.rs Resolved
@xiasongh
xiasongh requested a review from blp November 10, 2026 05:02
@mythical-fred-oss

Copy link
Copy Markdown
Contributor

LGTM. No feldera-pr-checks gate fails.

What I ran

Check Result
cargo test -p dbsp_connectors --lib idle_step_timer ✅ pass
cargo test -p static-connectors --test connector_suites -- idle_ ✅ pass (5.1 s)
Mutation: I commented out controller.request_step() ✅ idle_pipeline_steps_under_memory_pressure fails, so the test catches the regression
rustfmt --check on changed files ✅ clean

Missing (medium): no negative test. No test checks that a pipeline with no memory pressure stays idle. I wrote one and ran it locally, and it passes (10 s). Consider adding it, because it guards against the trigger firing without memory pressure (for example, if someone loosens the >= High check):

+/// An idle pipeline without memory pressure takes no idle step.
+#[engine_test]
+pub fn idle_pipeline_does_not_step_without_memory_pressure(engine: &dyn PipelineTestEngine) {
+    init_test_logger();
+    let config: PipelineConfig = serde_json::from_value(json!({
+        "name": "test", "workers": 1, "clock_resolution_usecs": null, "inputs": {}
+    })).unwrap();
+    let controller = engine
+        .controller(&identity(&[None]), &config, Box::new(|e, _| panic!("error: {e}")))
+        .unwrap();
+    controller.start();
+    let steps = || controller.status().global_metrics.total_initiated_steps();
+    let before = steps();
+    std::thread::sleep(IDLE_STEP_INTERVAL * 2);
+    assert_eq!(steps(), before);
+    controller.stop().unwrap();
+}

@xiasongh
xiasongh force-pushed the song/idle-step-memory-pressure-70ff87 branch 2 times, most recently from 8accdc2 to d62c48b Compare November 10, 2026 14:24
Comment thread crates/connectors/src/controller.rs Resolved
The ad-hoc snapshot keeps the last step's batches in memory, even after a
merge or a spill replaces them, and an idle pipeline takes no step to update
it. Under high memory pressure, request a step after 5 s with no step, at
most once each 5 s.
@xiasongh
xiasongh force-pushed the song/idle-step-memory-pressure-70ff87 branch from d62c48b to 0de5c4b Compare November 10, 2026 20:09
@xiasongh
xiasongh added this pull request to the merge queue Nov 10, 2026
Merged via the queue into main with commit 1844b29 Nov 11, 2026
1 check passed
@xiasongh
xiasongh deleted the song/idle-step-memory-pressure-70ff87 branch November 11, 2026 00:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants