Repository navigation
[connectors] Step an idle circuit under memory pressure - #7909
Conversation
|
What I ran
Gates: tests ✅. Flakiness: no flakiness seen. The wait has a 10 s margin over the 1 s stats tick, which looks safe. No connector or format change, no Tests the PR doesn't cover (medium):
|
|
missing (medium): negative test. I ran this locally: it passes on the PR and fails when the diff --git a/crates/connectors/src/controller/test.rs b/crates/connectors/src/controller/test.rs
index 9dc538b..3949956 100644
--- a/crates/connectors/src/controller/test.rs
+++ b/crates/connectors/src/controller/test.rs
@@ -11325,3 +11325,35 @@ pub fn paused_connector_holds_transaction(engine: &dyn PipelineTestEngine) {
assert!(paused().is_empty());
controller.stop().unwrap();
}
+
+/// Without memory pressure, an idle pipeline takes no step.
+#[engine_test]
+pub fn idle_pipeline_does_not_step_without_memory_pressure(engine: &dyn PipelineTestEngine) {
+ init_test_logger();
+
+ let config: PipelineConfig = serde_json::from_value(json!({
+ "name": "test",
+ "workers": 1,
+ "clock_resolution_usecs": null,
+ "inputs": {}
+ }))
+ .unwrap();
+ let controller = engine
+ .controller(
+ &identity(&[None]),
+ &config,
+ Box::new(|e, _| panic!("error: {e}")),
+ )
+ .unwrap();
+ controller.start();
+
+ let steps = || controller.status().global_metrics.total_initiated_steps();
+ let before = steps();
+ let timeout = IDLE_STEP_INTERVAL + Duration::from_secs(3);
+ assert!(
+ wait(|| steps() > before, timeout.as_millis() as u64).is_err(),
+ "an idle pipeline without memory pressure stepped"
+ );
+
+ controller.stop().unwrap();
+} |
|
LGTM. No feldera-pr-checks gate fails. What I ran
Missing (medium): no negative test. No test checks that a pipeline with no memory pressure stays idle. I wrote one and ran it locally, and it passes (10 s). Consider adding it, because it guards against the trigger firing without memory pressure (for example, if someone loosens the +/// An idle pipeline without memory pressure takes no idle step.
+#[engine_test]
+pub fn idle_pipeline_does_not_step_without_memory_pressure(engine: &dyn PipelineTestEngine) {
+ init_test_logger();
+ let config: PipelineConfig = serde_json::from_value(json!({
+ "name": "test", "workers": 1, "clock_resolution_usecs": null, "inputs": {}
+ })).unwrap();
+ let controller = engine
+ .controller(&identity(&[None]), &config, Box::new(|e, _| panic!("error: {e}")))
+ .unwrap();
+ controller.start();
+ let steps = || controller.status().global_metrics.total_initiated_steps();
+ let before = steps();
+ std::thread::sleep(IDLE_STEP_INTERVAL * 2);
+ assert_eq!(steps(), before);
+ controller.stop().unwrap();
+} |
8accdc2 to
d62c48b
Compare
The ad-hoc snapshot keeps the last step's batches in memory, even after a merge or a spill replaces them, and an idle pipeline takes no step to update it. Under high memory pressure, request a step after 5 s with no step, at most once each 5 s.
d62c48b to
0de5c4b
Compare
The ad-hoc snapshot keeps the last step's batches in memory, even after a merge or a spill replaces them (
SpineSnapshot,update_snapshot). An idle pipeline takes no step, so this memory stays in use. In a k3d test, anon memory stayed at 2.40 GB after a spill, and one step dropped it to 0.61 GB.Under high memory pressure, the statistics thread requests a step after 5 s with no step, at most once each 5 s. Only when no step runs and no transaction is open. The step goes through the existing
request_step()path.Tests:
idle_step_timer: the delay, the rate limit, and the restart after a step.idle_pipeline_steps_under_memory_pressure: with a 1 MB budget and no input, the pipeline steps.idle_pipeline_does_not_step_without_memory_pressure: without pressure, it does not.