← 输入框|github.com|刷新
Skip to content

[connectors] Limit the Samply profile duration and marker memory - #7826

Merged
blp merged 4 commits into
mainfrom
samply-duration-limit
Nov 5, 2026
Merged

blp merged 4 commits into
mainfrom
samply-duration-limit

Conversation

@blp

@blp blp commented Nov 4, 2026

Copy link
Copy Markdown
Member

Problem

A pentest found that POST /v0/pipelines/{name}/samply_profile accepts any duration_secs. A caller with the read role started a profile with duration_secs=2592000 (30 days) and got 202 Accepted.

A long duration has these costs:

Cost Location
Marker memory cap grows as workers × 8,000 markers/s × Span::BYTES × seconds: about 18 GB at 8 workers and 3,600 s crates/connectors/src/controller.rs
The profile file is read whole into memory crates/connectors/src/controller.rs
No other profile can start (409) while one runs crates/connectors/src/server.rs
A value above about 9.2 × 10^15 makes chrono::Duration::seconds panic in the handler crates/connectors/src/server.rs

Changes

  • The pipeline refuses duration_secs outside 1 to 300 with 400 InvalidParam, before it starts Samply. The check is in the pipeline because the manager forwards the query string unchanged.
  • Marker memory is capped at 1 GiB, whatever the worker count. Markers past the cap are dropped (crates/samply/src/lib.rs), so this does not cause an error.
  • The web console maximum goes from 3,600 to 300 s. Its largest preset was already 300 s.
  • The API docs (openapi.json, types.gen.ts) and the Python docstrings say "from 1 to 300".

I chose 400 over clamping silently: the web console computes its countdown from the duration it sends, so a silent clamp would make it wait for nothing.

openapi.json and types.gen.ts were edited by hand with the text the generators produce from the new doc comment. If the API client check in CI disagrees, run pre-commit run update-api-client --all-files.

Test

  • New test samply_profile_rejects_out_of_range_duration sends 0, 301, 2,592,000 and u64::MAX and expects 400 InvalidParam.
  • With the range check disabled, the test fails (the pipeline answers 500). With the check, it passes.

cargo test -p static-connectors --test connector_suites -- samply_profile_rejects

blp added 2 commits November 3, 2026 16:18
`POST /samply_profile` accepted any `duration_secs`.  A caller with the
read role could hold the profiler for 30 days, and a very large value
made the handler panic in `chrono::Duration::seconds`.

The pipeline now refuses a duration outside 1 to 300 seconds with 400
`InvalidParam`, before it starts Samply.  The web console maximum, the
API docs and the Python docstrings now say the same.

Signed-off-by: Ben Pfaff <blp@feldera.com>
The marker memory limit grows with the number of workers and the
profile duration.  With 64 workers and 300 seconds it is about 12 GB.
Markers past the limit are dropped, so a fixed ceiling is safe.

Signed-off-by: Ben Pfaff <blp@feldera.com>
@blp blp self-assigned this Nov 4, 2026
@blp
blp requested a review from mihaibudiu November 4, 2026 00:20
@blp blp added API Distributed system APIs connectors Issues related to the adapters/connectors crate Good First Issue rust Pull requests that update Rust code profiler Issues related to the profiler and its APIs python Pull requests that update python code metrics Metrics about feldera pipelines and removed Good First Issue labels Nov 4, 2026
let body = response.body().await.unwrap();
let error: serde_json::Value = serde_json::from_slice(&body).unwrap();
assert_eq!(error["error_code"], "InvalidParam", "{duration}: {error}");
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

missing (medium): this only checks values that should be rejected. Nothing confirms the boundaries 1 and 300 are still accepted, so an off-by-one such as 1..MAX would pass. I tried the addition below locally and it passes in about 1.5 s:

                 assert_eq!(error["error_code"], "InvalidParam", "{duration}: {error}");
             }
+
+            // The boundaries pass the range check, whether or not Samply then starts.
+            for duration in [1, max] {
+                let response = server
+                    .post(format!("/samply_profile?duration_secs={duration}"))
+                    .send()
+                    .await
+                    .unwrap();
+                assert_ne!(response.status(), StatusCode::BAD_REQUEST, "{duration}");
+            }

@mythical-fred-oss

Copy link
Copy Markdown
Contributor

What I ran: cargo test --locked -p static-connectors --test connector_suites -- samply_profile_rejects passes. The boundary check I suggested inline also passes.

Label Severity Finding
missing (API docs) medium The #[utoipa::path] on start_samply_profile (crates/pipeline-manager/src/api/endpoints/pipeline_interaction.rs) lists no BAD_REQUEST response, so the 400 InvalidParam this PR adds is missing from openapi.json. Add the response there and regenerate the spec instead of editing it by hand.
missing (docs gate) medium docs.feldera.com/docs/operations/visualizing-profiles.md has a "Failures" section for the samply endpoint. It should say that duration_secs must be from 1 to 300 and that other values get a 400. This is a user-visible API change.
pitfall (breaking change) low-medium Before this PR, the web console and the API accepted up to 3,600 s. Scripts that ask for more than 300 s will now get a 400. This deserves a line in the release notes or changelog.

Apart from these, the range check, the 1 GiB marker memory cap and the commit messages look good to me.

/// In a multihost pipeline, the ordinal of the pipeline to sample.
pub ordinal: usize,
/// The number of seconds to sample for the profile.
/// The number of seconds to sample for the profile, from 1 to 300.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would say from 1 to MAX_DURATION_SECS
But maybe this impacts openapi.json?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, it would affect openapi.json. This doc comment becomes the duration_secs description there and in types.gen.ts. utoipa copies the text as written and cannot put in the value of the constant, so API users would see the name MAX_DURATION_SECS, which they cannot look up. For this reason I kept "300" in the text. The handler and the test use the constant, so the code has only one copy of the number.

-- Claude Code, replying at @blp's request.


:param pipeline_name: The name of the pipeline
:param duration: The duration of the profile in seconds (default: 30)
:param duration: The duration of the profile in seconds, from 1 to 300 (default: 30)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it important to say the max value here?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it is useful. The server refuses values above 300 with a 400, and the docstring is the only place where a Python user can see the limit before that happens. I kept it.

-- Claude Code, replying at @blp's request.

blp added 2 commits November 4, 2026 13:44
The test checked only durations that the pipeline must refuse, so an
off-by-one error in the range check would not cause a failure.

Signed-off-by: Ben Pfaff <blp@feldera.com>
`POST /v0/pipelines/{name}/samply_profile` refuses a `duration_secs`
outside 1 to 300 with 400 `InvalidParam`.  The OpenAPI spec and the
profiling docs now say so.

Signed-off-by: Ben Pfaff <blp@feldera.com>
@blp
blp added this pull request to the merge queue Nov 4, 2026
Merged via the queue into main with commit f67d1c8 Nov 5, 2026
1 check passed
@blp
blp deleted the samply-duration-limit branch November 5, 2026 04:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

API Distributed system APIs connectors Issues related to the adapters/connectors crate metrics Metrics about feldera pipelines profiler Issues related to the profiler and its APIs python Pull requests that update python code rust Pull requests that update Rust code

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants