Repository navigation
[pipeline-manager] Make the JSON request body limit configurable - #7916
Conversation
|
@anandbraman why not raise the limit a lot by default? We can obviously lower the limit for things like the sandbox. |
|
@lalithsuresh what do you think a sensible default is? |
|
LGTM. All feldera-pr-checks gates pass: it has a test, docs and a changelog entry, adds no new dependencies, and has no What I ran
Missing (low-medium): test coverage. The PR's test covers only PUT with let req = test::TestRequest::put()
.uri("/v0/pipelines/p")
.insert_header(header::ContentType::json())
.set_payload(serde_json::to_vec(&body).unwrap()) // no Content-Length
.to_request();
let resp = test::call_service(&app, req).await;
assert_eq!(resp.status(), 413); // body contains FELDERA_MAX_JSON_BODY_BYTES |
927a83f to
87b46c9
Compare
|
What I ran
missing (medium): the PR's tests don't cover compressed bodies or the CLI/env parsing. The tests I ran for both are below. Please consider adding them: tests--- a/crates/pipeline-manager/src/config.rs
+ #[test]
+ fn max_json_body_bytes_parses() {
+ let unset = ApiServerConfig::try_parse_from(["api-server"]).unwrap();
+ assert_eq!(unset.max_json_body_bytes, 64 * 1024 * 1024);
+ let config =
+ ApiServerConfig::try_parse_from(["api-server", "--max-json-body-bytes=1048576"])
+ .unwrap();
+ assert_eq!(config.max_json_body_bytes, 1048576);
+ assert!(ApiServerConfig::try_parse_from(["api-server", "--max-json-body-bytes=64MiB"]).is_err());
+ assert!(ApiServerConfig::try_parse_from(["api-server", "--max-json-body-bytes=-1"]).is_err());
+ }
--- a/crates/pipeline-manager/src/api/main.rs
+ #[actix_web::test]
+ async fn max_json_body_bytes_counts_decompressed_size() {
+ // ... same setup as max_json_body_bytes_limits_pipeline_put ...
+ let mut enc = GzEncoder::new(Vec::new(), Compression::default());
+ enc.write_all(&serde_json::to_vec(&body).unwrap()).unwrap();
+ let gz = enc.finish().unwrap();
+ cfg.max_json_body_bytes = 2 * 1024 * 1024;
+ let req = test::TestRequest::post().uri("/v0/pipelines")
+ .insert_header((header::CONTENT_TYPE, "application/json"))
+ .insert_header((header::CONTENT_ENCODING, "gzip"))
+ .set_payload(gz).to_request();
+ // expect 413 with FELDERA_MAX_JSON_BODY_BYTES in the body
+ }commit history (medium): the first commit says "The default stays 2 MiB", and the second commit changes it to 64 MiB. Please squash them before merge so The gates for tests, flakiness (no sleeps, deterministic), docs and the changelog are met. There is no breaking change: the limit only goes up, and the 413 status is unchanged. |
1168388 to
4a524b2
Compare
5b091b0 to
1fe57a1
Compare
The API server refused a pipeline body over 2 MiB, the actix-web default, and no setting could change it. Generated SQL programs can be larger than that, so they could not be submitted. --max-json-body-bytes (FELDERA_MAX_JSON_BODY_BYTES) now sets the limit. The default stays 2 MiB. A body over the limit still gets 413, and the message now names the setting. A new test sends a 3 MiB pipeline with the default limit and with a larger one. The docs and the changelog describe the setting. Fixes #7913 Signed-off-by: Anand Raman <anand.raman@feldera.com>
With a 2 MiB default, large generated SQL programs still needed a setting before they could be submitted. The default is now 64 MiB. Deployments with little memory, such as a sandbox, can lower it with --max-json-body-bytes. The test now checks a 3 MiB pipeline against a 2 MiB limit and against the default. Signed-off-by: Anand Raman <anand.raman@feldera.com>
Without a Content-Length header, actix-web finds the limit while the body streams in and returns JsonPayloadError::Overflow, not OverflowKnownLength. The test now sends such a body too, so the Overflow arm of the error handler is covered. Signed-off-by: Anand Raman <anand.raman@feldera.com>
Review asked what a "pipeline" body is. The setting and the docs now list what a pipeline request body holds: name, description, tags, SQL program, Rust UDF code, and runtime, program and deployment configuration. They also name the diff and validate_program endpoints, and say plainly that the limit caps the memory one request can use, because the server holds each body in memory while it parses it. Signed-off-by: Anand Raman <anand.raman@feldera.com>
The test now also sends a gzip body that is small on the wire but over the limit once decompressed, and expects 413 with the setting named. A new test checks that --max-json-body-bytes defaults to 64 MiB, takes a plain number of bytes, and refuses "64MiB" and "-1". The flag's doc comment now says that the limit applies to every JSON endpoint. Signed-off-by: Anand Raman <anand.raman@feldera.com>
…iler limit A JSON body over the limit now gets the standard ErrorResponse body with error code JsonBodyTooLarge. Before, the body was plain text, so the web console showed "Request failed" and fda printed a version warning, and neither showed the setting to raise. The compiler server read /validate_program with the actix-web default limit of 2 MiB. Programs that the API server accepted then failed validation and diff with 503. The API server already applies --max-json-body-bytes, so the compiler server has no second limit. Signed-off-by: Anand Raman <anand.raman@feldera.com>
5f74b7e to
46cb81b
Compare
| /// server, which already applied its own limit (`--max-json-body-bytes`), so there is no second | ||
| /// limit here. The actix-web default of 2 MiB would reject programs that the API server accepts. | ||
| fn json_config() -> web::JsonConfig { | ||
| web::JsonConfig::default().limit(usize::MAX) |
There was a problem hiding this comment.
pitfall (medium): usize::MAX removes any bound on the compiler server. When require_client_certificates is off (the default), nothing authenticates callers. A pipeline pod, which runs user Rust UDFs and can reach this port to fetch binaries, can stream an unbounded body to /validate_program and OOM the compiler server. Before this change, the actix-web default capped it at 2 MiB. Suggest a finite cap: pass the same --max-json-body-bytes value through CommonConfig, or use a fixed generous constant (e.g. 2× the API default).
|
What I ran
Notes
for req in [
test::TestRequest::post().uri("/v0/pipelines").set_json(&big).to_request(),
test::TestRequest::patch().uri("/v0/pipelines/p").set_json(&big).to_request(),
] {
assert_eq!(test::call_service(&app, req).await.status(), 413);
}
// a body of exactly `max_json_body_bytes` bytes gets 201
Gates: tests ✅, flakiness ✅ (no sleeps or timing), docs/changelog ✅, no new deps, no unsafe. I'd fix the unbounded compiler-server limit (inline comment) before merge. Otherwise it looks good. |
Fixes #7913.
The API server refused a pipeline body over 2 MiB, the actix-web default for
web::Json, and no setting could change it. This adds--max-json-body-bytes(FELDERA_MAX_JSON_BODY_BYTES) and raises the default to 64 MiB, so large generated SQL programs work without a setting. Deployments with little memory, such as the sandbox, can lower it.build_appregisters aweb::JsonConfigwith the configured limit. It coversPOST,PUTandPATCHof/v0/pipelines, and the other JSON endpoints.controlPlane.env), Docker guide, operations guide, changelog.Testing
max_json_body_bytes_limits_pipeline_put: a 3 MiB pipeline gets 413 with a 2 MiB limit, sent withContent-Lengthand streamed without it, and 201 with the default limit. With theJsonConfigline removed, or theOverflowarm of the error handler, the test fails.cargo test -p pipeline-manager --lib -- api::main auth:: config::, clippy with-D warnings, and the pre-commit hooks pass.PUTgets 413 with a 2 MiB limit. With a 64 MiB limit, 2.2 MB and 29 MBPUTs get 200.AI use
Claude Code wrote the change, the test and the docs, and ran the checks above.