Repository navigation
[connectors] Limit the Samply profile duration and marker memory - #7826
Conversation
`POST /samply_profile` accepted any `duration_secs`. A caller with the read role could hold the profiler for 30 days, and a very large value made the handler panic in `chrono::Duration::seconds`. The pipeline now refuses a duration outside 1 to 300 seconds with 400 `InvalidParam`, before it starts Samply. The web console maximum, the API docs and the Python docstrings now say the same. Signed-off-by: Ben Pfaff <blp@feldera.com>
The marker memory limit grows with the number of workers and the profile duration. With 64 workers and 300 seconds it is about 12 GB. Markers past the limit are dropped, so a fixed ceiling is safe. Signed-off-by: Ben Pfaff <blp@feldera.com>
| let body = response.body().await.unwrap(); | ||
| let error: serde_json::Value = serde_json::from_slice(&body).unwrap(); | ||
| assert_eq!(error["error_code"], "InvalidParam", "{duration}: {error}"); | ||
| } |
There was a problem hiding this comment.
missing (medium): this only checks values that should be rejected. Nothing confirms the boundaries 1 and 300 are still accepted, so an off-by-one such as 1..MAX would pass. I tried the addition below locally and it passes in about 1.5 s:
assert_eq!(error["error_code"], "InvalidParam", "{duration}: {error}");
}
+
+ // The boundaries pass the range check, whether or not Samply then starts.
+ for duration in [1, max] {
+ let response = server
+ .post(format!("/samply_profile?duration_secs={duration}"))
+ .send()
+ .await
+ .unwrap();
+ assert_ne!(response.status(), StatusCode::BAD_REQUEST, "{duration}");
+ }|
What I ran:
Apart from these, the range check, the 1 GiB marker memory cap and the commit messages look good to me. |
| /// In a multihost pipeline, the ordinal of the pipeline to sample. | ||
| pub ordinal: usize, | ||
| /// The number of seconds to sample for the profile. | ||
| /// The number of seconds to sample for the profile, from 1 to 300. |
There was a problem hiding this comment.
I would say from 1 to MAX_DURATION_SECS
But maybe this impacts openapi.json?
There was a problem hiding this comment.
Yes, it would affect openapi.json. This doc comment becomes the duration_secs description there and in types.gen.ts. utoipa copies the text as written and cannot put in the value of the constant, so API users would see the name MAX_DURATION_SECS, which they cannot look up. For this reason I kept "300" in the text. The handler and the test use the constant, so the code has only one copy of the number.
-- Claude Code, replying at @blp's request.
|
|
||
| :param pipeline_name: The name of the pipeline | ||
| :param duration: The duration of the profile in seconds (default: 30) | ||
| :param duration: The duration of the profile in seconds, from 1 to 300 (default: 30) |
There was a problem hiding this comment.
Is it important to say the max value here?
There was a problem hiding this comment.
I think it is useful. The server refuses values above 300 with a 400, and the docstring is the only place where a Python user can see the limit before that happens. I kept it.
-- Claude Code, replying at @blp's request.
The test checked only durations that the pipeline must refuse, so an off-by-one error in the range check would not cause a failure. Signed-off-by: Ben Pfaff <blp@feldera.com>
`POST /v0/pipelines/{name}/samply_profile` refuses a `duration_secs`
outside 1 to 300 with 400 `InvalidParam`. The OpenAPI spec and the
profiling docs now say so.
Signed-off-by: Ben Pfaff <blp@feldera.com>
Problem
A pentest found that
POST /v0/pipelines/{name}/samply_profileaccepts anyduration_secs. A caller with the read role started a profile withduration_secs=2592000(30 days) and got202 Accepted.A long duration has these costs:
Span::BYTES× seconds: about 18 GB at 8 workers and 3,600 scrates/connectors/src/controller.rscrates/connectors/src/controller.rscrates/connectors/src/server.rschrono::Duration::secondspanic in the handlercrates/connectors/src/server.rsChanges
duration_secsoutside 1 to 300 with 400InvalidParam, before it starts Samply. The check is in the pipeline because the manager forwards the query string unchanged.crates/samply/src/lib.rs), so this does not cause an error.openapi.json,types.gen.ts) and the Python docstrings say "from 1 to 300".I chose 400 over clamping silently: the web console computes its countdown from the duration it sends, so a silent clamp would make it wait for nothing.
openapi.jsonandtypes.gen.tswere edited by hand with the text the generators produce from the new doc comment. If the API client check in CI disagrees, runpre-commit run update-api-client --all-files.Test
samply_profile_rejects_out_of_range_durationsends 0, 301, 2,592,000 andu64::MAXand expects 400InvalidParam.cargo test -p static-connectors --test connector_suites -- samply_profile_rejects