← 输入框|github.com|刷新
Skip to content

[pipeline-manager] Make the JSON request body limit configurable - #7916

Merged
anandbraman merged 6 commits into
mainfrom
anandraman/configurable-json-body-limit
Nov 11, 2026
Merged

anandbraman merged 6 commits into
mainfrom
anandraman/configurable-json-body-limit

Conversation

@anandbraman

@anandbraman anandbraman commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #7913.

The API server refused a pipeline body over 2 MiB, the actix-web default for web::Json, and no setting could change it. This adds --max-json-body-bytes (FELDERA_MAX_JSON_BODY_BYTES) and raises the default to 64 MiB, so large generated SQL programs work without a setting. Deployments with little memory, such as the sandbox, can lower it.

  • build_app registers a web::JsonConfig with the configured limit. It covers POST, PUT and PATCH of /v0/pipelines, and the other JSON endpoints.
  • A body over the limit still gets 413. The message now names the setting.
  • Docs: Helm chart reference (through controlPlane.env), Docker guide, operations guide, changelog.

Testing

  • New test max_json_body_bytes_limits_pipeline_put: a 3 MiB pipeline gets 413 with a 2 MiB limit, sent with Content-Length and streamed without it, and 201 with the default limit. With the JsonConfig line removed, or the Overflow arm of the error handler, the test fails.
  • cargo test -p pipeline-manager --lib -- api::main auth:: config::, clippy with -D warnings, and the pre-commit hooks pass.
  • Manual, with an earlier version of this change (environment variable only) on v0.381.0: a 2.2 MB PUT gets 413 with a 2 MiB limit. With a 64 MiB limit, 2.2 MB and 29 MB PUTs get 200.

AI use

Claude Code wrote the change, the test and the docs, and ran the checks above.

@lalithsuresh

Copy link
Copy Markdown
Contributor

@anandbraman why not raise the limit a lot by default? We can obviously lower the limit for things like the sandbox.

@anandbraman

Copy link
Copy Markdown
Contributor Author

@lalithsuresh what do you think a sensible default is?

@mythical-fred-oss

Copy link
Copy Markdown
Contributor

LGTM. All feldera-pr-checks gates pass: it has a test, docs and a changelog entry, adds no new dependencies, and has no unsafe code.

What I ran

Check Result
cargo test --locked -p pipeline-manager --lib -- max_json_body_bytes ✅ pass
pre-commit run --files <changed files> (clippy, fmt, OpenAPI, docs) ✅ pass
QA test I wrote locally (not pushed): PATCH over the limit, a body with no Content-Length header (Overflow branch), malformed JSON ✅ 413 / 413 with the setting named / 400

Missing (low-medium): test coverage. The PR's test covers only PUT with Content-Length, which is the OverflowKnownLength branch. Consider adding the streaming case, which exercises the Overflow arm of the error handler. It passes today:

let req = test::TestRequest::put()
    .uri("/v0/pipelines/p")
    .insert_header(header::ContentType::json())
    .set_payload(serde_json::to_vec(&body).unwrap()) // no Content-Length
    .to_request();
let resp = test::call_service(&app, req).await;
assert_eq!(resp.status(), 413); // body contains FELDERA_MAX_JSON_BODY_BYTES

@anandbraman
anandbraman requested a review from gz November 10, 2026 19:12
@anandbraman
anandbraman marked this pull request as ready for review November 10, 2026 19:12
@anandbraman
anandbraman force-pushed the anandraman/configurable-json-body-limit branch from 927a83f to 87b46c9 Compare November 10, 2026 19:12
Comment thread crates/pipeline-manager/src/config.rs Resolved
Comment thread crates/pipeline-manager/src/config.rs Resolved
Comment thread crates/pipeline-manager/src/api/main.rs Resolved
@mythical-fred-oss

Copy link
Copy Markdown
Contributor

What I ran

Check Result
cargo test --locked -p pipeline-manager --lib -- max_json_body_bytes ✅ pass
pre-commit run on the PR's files (clippy, fmt, machete, OpenAPI) ✅ pass
QA: gzip body, 3 MiB inflated but under 64 KiB on the wire, POST /v0/pipelines with a 2 MiB limit ✅ 413, and the message names the setting (so the limit is counted after decompression, as the docs say)
QA: clap parsing of the flag (default 64 MiB, =1048576, rejects 64MiB and -1) ✅ pass

missing (medium): the PR's tests don't cover compressed bodies or the CLI/env parsing. The tests I ran for both are below. Please consider adding them:

tests
--- a/crates/pipeline-manager/src/config.rs
+    #[test]
+    fn max_json_body_bytes_parses() {
+        let unset = ApiServerConfig::try_parse_from(["api-server"]).unwrap();
+        assert_eq!(unset.max_json_body_bytes, 64 * 1024 * 1024);
+        let config =
+            ApiServerConfig::try_parse_from(["api-server", "--max-json-body-bytes=1048576"])
+                .unwrap();
+        assert_eq!(config.max_json_body_bytes, 1048576);
+        assert!(ApiServerConfig::try_parse_from(["api-server", "--max-json-body-bytes=64MiB"]).is_err());
+        assert!(ApiServerConfig::try_parse_from(["api-server", "--max-json-body-bytes=-1"]).is_err());
+    }
--- a/crates/pipeline-manager/src/api/main.rs
+    #[actix_web::test]
+    async fn max_json_body_bytes_counts_decompressed_size() {
+        // ... same setup as max_json_body_bytes_limits_pipeline_put ...
+        let mut enc = GzEncoder::new(Vec::new(), Compression::default());
+        enc.write_all(&serde_json::to_vec(&body).unwrap()).unwrap();
+        let gz = enc.finish().unwrap();
+        cfg.max_json_body_bytes = 2 * 1024 * 1024;
+        let req = test::TestRequest::post().uri("/v0/pipelines")
+            .insert_header((header::CONTENT_TYPE, "application/json"))
+            .insert_header((header::CONTENT_ENCODING, "gzip"))
+            .set_payload(gz).to_request();
+        // expect 413 with FELDERA_MAX_JSON_BODY_BYTES in the body
+    }

commit history (medium): the first commit says "The default stays 2 MiB", and the second commit changes it to 64 MiB. Please squash them before merge so main has no commit message that later turns out to be false.

The gates for tests, flakiness (no sleeps, deterministic), docs and the changelog are met. There is no breaking change: the limit only goes up, and the 413 status is unchanged.

@anandbraman
anandbraman force-pushed the anandraman/configurable-json-body-limit branch from 1168388 to 4a524b2 Compare November 10, 2026 22:53
@anandbraman
anandbraman disabled auto-merge November 11, 2026 03:46
@anandbraman
anandbraman force-pushed the anandraman/configurable-json-body-limit branch 2 times, most recently from 5b091b0 to 1fe57a1 Compare November 11, 2026 15:27
The API server refused a pipeline body over 2 MiB, the actix-web
default, and no setting could change it. Generated SQL programs can be
larger than that, so they could not be submitted.

--max-json-body-bytes (FELDERA_MAX_JSON_BODY_BYTES) now sets the limit.
The default stays 2 MiB. A body over the limit still gets 413, and the
message now names the setting. A new test sends a 3 MiB pipeline with
the default limit and with a larger one. The docs and the changelog
describe the setting.

Fixes #7913

Signed-off-by: Anand Raman <anand.raman@feldera.com>
With a 2 MiB default, large generated SQL programs still needed a
setting before they could be submitted. The default is now 64 MiB.
Deployments with little memory, such as a sandbox, can lower it with
--max-json-body-bytes. The test now checks a 3 MiB pipeline against a
2 MiB limit and against the default.

Signed-off-by: Anand Raman <anand.raman@feldera.com>
Without a Content-Length header, actix-web finds the limit while the
body streams in and returns JsonPayloadError::Overflow, not
OverflowKnownLength. The test now sends such a body too, so the
Overflow arm of the error handler is covered.

Signed-off-by: Anand Raman <anand.raman@feldera.com>
Review asked what a "pipeline" body is. The setting and the docs now
list what a pipeline request body holds: name, description, tags, SQL
program, Rust UDF code, and runtime, program and deployment
configuration. They also name the diff and validate_program endpoints,
and say plainly that the limit caps the memory one request can use,
because the server holds each body in memory while it parses it.

Signed-off-by: Anand Raman <anand.raman@feldera.com>
The test now also sends a gzip body that is small on the wire but over
the limit once decompressed, and expects 413 with the setting named. A
new test checks that --max-json-body-bytes defaults to 64 MiB, takes a
plain number of bytes, and refuses "64MiB" and "-1". The flag's doc
comment now says that the limit applies to every JSON endpoint.

Signed-off-by: Anand Raman <anand.raman@feldera.com>
…iler limit

A JSON body over the limit now gets the standard ErrorResponse body
with error code JsonBodyTooLarge. Before, the body was plain text, so
the web console showed "Request failed" and fda printed a version
warning, and neither showed the setting to raise.

The compiler server read /validate_program with the actix-web default
limit of 2 MiB. Programs that the API server accepted then failed
validation and diff with 503. The API server already applies
--max-json-body-bytes, so the compiler server has no second limit.

Signed-off-by: Anand Raman <anand.raman@feldera.com>
@anandbraman
anandbraman force-pushed the anandraman/configurable-json-body-limit branch from 5f74b7e to 46cb81b Compare November 11, 2026 20:55
@anandbraman
anandbraman added this pull request to the merge queue Nov 11, 2026
/// server, which already applied its own limit (`--max-json-body-bytes`), so there is no second
/// limit here. The actix-web default of 2 MiB would reject programs that the API server accepts.
fn json_config() -> web::JsonConfig {
web::JsonConfig::default().limit(usize::MAX)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pitfall (medium): usize::MAX removes any bound on the compiler server. When require_client_certificates is off (the default), nothing authenticates callers. A pipeline pod, which runs user Rust UDFs and can reach this port to fetch binaries, can stream an unbounded body to /validate_program and OOM the compiler server. Before this change, the actix-web default capped it at 2 MiB. Suggest a finite cap: pass the same --max-json-body-bytes value through CommonConfig, or use a fixed generous constant (e.g. 2× the API default).

@mythical-fred-oss

Copy link
Copy Markdown
Contributor

What I ran

Check Result
cargo test -p pipeline-manager --lib -- max_json_body_bytes validate_program_accepts_large_program 3/3 pass
pre-commit run --files <changed> (clippy, fmt, OpenAPI, docs) pass
Extra QA test, not committed: limit 1024; POST /v0/pipelines and PATCH /v0/pipelines/p over the limit → 413 JsonBodyTooLarge; malformed JSON → 400; body exactly at the limit → 201 pass

Notes

  • missing (medium): the PR's test only covers PUT. POST, PATCH and the exact-limit boundary aren't covered, though my QA run shows they behave correctly. Worth adding the cases below:
for req in [
    test::TestRequest::post().uri("/v0/pipelines").set_json(&big).to_request(),
    test::TestRequest::patch().uri("/v0/pipelines/p").set_json(&big).to_request(),
] {
    assert_eq!(test::call_service(&app, req).await.status(), 413);
}
// a body of exactly `max_json_body_bytes` bytes gets 201
  • pitfall (medium): the default limit goes up 32× (2 → 64 MiB). Each request holds the raw buffer and the parsed String at once. With auth disabled, anyone who can reach the API can send several of these at the same time. The docs say this, so I'm only flagging it as a deliberate tradeoff for operators.
  • tech-debt (low-medium): commit 1 says "The default stays 2 MiB", and commit 2 then changes that. Commit 4 starts with "Review asked…". Please squash or reword these before merge, since main keeps a linear history.

Gates: tests ✅, flakiness ✅ (no sleeps or timing), docs/changelog ✅, no new deps, no unsafe. I'd fix the unbounded compiler-server limit (inline comment) before merge. Otherwise it looks good.

Merged via the queue into main with commit bf82b61 Nov 11, 2026
1 check passed
@anandbraman
anandbraman deleted the anandraman/configurable-json-body-limit branch November 11, 2026 22:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[pipeline-manager] Make the JSON request body limit configurable (pipeline create and update reject programs over 2 MiB)

3 participants