← 输入框|github.com|刷新
Skip to content

[pipeline-manager] Backport: close a connection when the client closes it (v0.334.3) - #7964

Closed
xiasongh wants to merge 3 commits into
v0.334.3from
song/backport-half-closed-v0.334.2
Closed

xiasongh wants to merge 3 commits into
v0.334.3from
song/backport-half-closed-v0.334.2

Conversation

@xiasongh

@xiasongh xiasongh commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Backport of #7963 (commit d4b18b8) to v0.334.2, for a customer whose kubernetes-runner memory grows until it is killed for being out of memory.

Problem

Each GET /v0/pipelines/{name}/logs that the client stops, on a pipeline that writes no new log lines, leaks one connection and about 50 KB in the API server and in the runner. Actix keeps a half-closed HTTP/1 connection open. A streaming response finds out that the client left only when a write fails, and a quiet logs stream does not write.

Change

File Change
Cargo.toml, Cargo.lock actix-web 4.11.0 -> 4.12.0 (first version where HttpServer has h1_allow_half_closed). Only the actix-web lockfile entry changes. actix-http is already 3.12.1.
crates/pipeline-manager/src/api/main.rs, runner/main.rs Call .h1_allow_half_closed(false) on each HttpServer, with a comment.

At v0.334.2 the runner HttpServer has no .on_connect(Caller::record), so only the new call is added there. cargo update -p actix-web --precise 4.12.0 also moved 19 unrelated edges (windows-sys, darling, socket2 for hyper-util and quinn), so the lockfile edit is limited to the actix-web entry. cargo metadata --locked accepts it with cargo 1.93.1 and 1.96.1.

The logs endpoint at this version is the older one (no cursor protocol). get_logs in the runner and http_streaming_logs_from_pipeline_by_name in the API server do not change. When the client closes, the API server drops its awc response to the runner, the runner drops the follower receiver, and the logs thread removes the closed follower on its idle cleanup tick.

The three commits end in the same state as the change on main: no constant, no http_server module, no test. Squash on merge.

Base branch v0.334.3 is a copy of v0.334.2 (aced2c37). No version bump and no tag in this PR.

A client that stops a GET /v0/pipelines/{name}/logs request leaked a
connection in the API server and in the runner, together with a log
follower in the runner. Actix keeps a half-closed HTTP/1 connection open,
and a streaming response finds out that the client left only when a write
to it fails. The logs stream of a pipeline that writes no new lines does
not write, so the connection stayed open until the process restarted.
Each stopped request kept about 50 KB and one file descriptor in each
process. A runner with a 512 MiB memory limit was killed for being out
of memory after some thousands of such requests.

The API server and the runner now close a connection when they read EOF
from the client. The API server then drops its request to the runner, the
runner drops the follower's channel, and the logs thread removes the
follower.

Actix-web 4.12 is the first version that lets HttpServer set this.

This is a backport to v0.334.2. At v0.334.2 the runner HTTP server has
no on_connect hook, so only the h1_allow_half_closed call is added there.
Only the actix-web entry changes in Cargo.lock.

(cherry picked from commit d4b18b8)
The API server and the runner now call h1_allow_half_closed(false)
directly, with a comment that tells why. The http_server module and its
H1_ALLOW_HALF_CLOSED constant are removed. The test that a client
disconnect drops a silent streaming body moves to a new test module in
runner/main.rs and sets h1_allow_half_closed(false) on its own server.
This matches the change on main.
The test started its own actix server with h1_allow_half_closed(false),
so it tested actix, not the API server or the runner. It is removed to
match the change on main.
@xiasongh

Copy link
Copy Markdown
Contributor Author

Closing: the backport now follows the make-a-bugfix-release skill. b67edbc and e688e60 from #7963 are cherry-picked directly onto the v0.334.3 branch, followed by the 0.334.3 version bump.

@xiasongh xiasongh closed this Nov 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant