← 输入框|github.com|刷新
Skip to content

[dbsp] Drop page cache so a lower memory limit can apply on resize - #7903

Merged
xiasongh merged 1 commit into
mainfrom
song/feldera-memory-limit-resize-401382
Nov 10, 2026
Merged

xiasongh merged 1 commit into
mainfrom
song/feldera-memory-limit-resize-401382

Conversation

@xiasongh

@xiasongh xiasongh commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

Kubernetes lowers a running pod's memory limit only when its memory use, page cache included, is below the new limit. Page cache can block a resize down forever.

  • On a resize, the RSS monitor drops storage files from the page cache (posix_fadvise(DONTNEED)) until the limit applies. Each pass stops after 500 ms; the drop stops after 30 min.
  • set_max_rss_bytes becomes set_memory_limit(max_rss, container_limit).
  • Incompatible: the memory budget is now min(max_rss, memory_max), also at startup. A datafusion_memory of memory_max or more now fails to start. See the changelog.

Tested on k3d: a 3G to 1500M resize held up by page cache applied in 12 s, with no restart.

@xiasongh
xiasongh force-pushed the song/feldera-memory-limit-resize-401382 branch from 9e2103f to cceb044 Compare November 9, 2026 20:05
@xiasongh
xiasongh marked this pull request as ready for review November 9, 2026 20:06
@xiasongh
xiasongh requested review from blp and gz November 9, 2026 20:14
Comment thread crates/dbsp/src/circuit/runtime.rs Resolved
Comment thread crates/feldera-types/src/config.rs Resolved
Comment thread crates/dbsp/src/storage/page_cache.rs Resolved
@mythical-fred-oss

Copy link
Copy Markdown
Contributor

What I ran (debug, at cceb044)

Check Result
cargo test -p feldera-observability ✅ 5/5
cargo test -p feldera-types effective_memory ✅
cargo test -p dbsp --lib page_cache max_rss ✅ 8/8; page_cache tests looped 30× on ext4 with no failures
cargo test -p static-connectors --test connector_suites resize_sets_memory_budget ✅
pre-commit run --files <changed> ✅

Gates

  • Docs / breaking change (medium): max_rss is now capped by memory_max at startup too, so some existing configs no longer start. The PR has no changelog entry, and docs/operations/memory.md and the max_rss rustdoc are stale. Details are inline. Please fix before merge.
  • Flaky tests (medium): see the inline comment on the hardcoded page size.

Missing test cases (medium)

  1. Nothing tests the wiring from set_memory_limit(_, Some(..)) to the RSS monitor: that it sets a Target, and that the next tick clears it once memory.max is already at or below the target. A resize up should leave no target behind.
  2. check() while a new target arrives during a drop: the since comparison that resets the counts is untested.
  3. Startup with max_rss > memory_max: only the resize path and the pure effective_memory function are tested.

@blp

blp commented Nov 9, 2026

Copy link
Copy Markdown
Member

I noticed that this avoids dropping files that are still mutable, but I think it's OK to drop those anyway because we write them sequentially.

@blp

blp commented Nov 9, 2026

Copy link
Copy Markdown
Member

I think there's a sysctl to drop the whole page cache. It might be easier.

@xiasongh

xiasongh commented Nov 9, 2026

Copy link
Copy Markdown
Contributor Author

I think there's a sysctl to drop the whole page cache. It might be easier.

do you mean /proc/sys/vm/drop_caches? i ruled that out since it drops entire system's page cache, which would affect other pods on the node

@blp

blp commented Nov 9, 2026

Copy link
Copy Markdown
Member
+- Kubernetes lowers a pod's memory limit only when the pod's memory use, page cache included, is
+  below the new limit. To get there, the pipeline drops its storage files from the page cache,
+  for at most 30 minutes. Reads from storage are slower until the cache fills again.

Everything involving storage might be slower permanently since we're caching less.

@blp blp left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a lot, but I don't see a problem with it. Thank you!

@xiasongh
xiasongh force-pushed the song/feldera-memory-limit-resize-401382 branch from cceb044 to 27df58a Compare November 10, 2026 02:47
Kubernetes lowers a pod's memory limit only when usage, page cache
included, is below it. On a resize, drop storage files from the page
cache until it is.

The memory budget is now min(max_rss, memory_max), also at startup.
@xiasongh
xiasongh force-pushed the song/feldera-memory-limit-resize-401382 branch from 27df58a to 330e986 Compare November 10, 2026 03:25
@xiasongh
xiasongh added this pull request to the merge queue Nov 10, 2026
Merged via the queue into main with commit aacdf91 Nov 10, 2026
1 check passed
@xiasongh
xiasongh deleted the song/feldera-memory-limit-resize-401382 branch November 10, 2026 05:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants