Repository navigation
Conversation
A client that stops a GET /v0/pipelines/{name}/logs request leaked a
connection in the API server and in the runner, together with a log
follower in the runner. Actix keeps a half-closed HTTP/1 connection open,
and a streaming response finds out that the client left only when a write
to it fails. The logs stream of a pipeline that writes no new lines does
not write, so the connection stayed open until the process restarted.
Each stopped request kept about 50 KB and one file descriptor in each
process. A runner with a 512 MiB memory limit was killed for being out
of memory after some thousands of such requests.
The API server and the runner now close a connection when they read EOF
from the client. The API server then drops its request to the runner, the
runner drops the follower's channel, and the logs thread removes the
follower.
Actix-web 4.12 is the first version that lets HttpServer set this.
This is a backport to v0.334.2. At v0.334.2 the runner HTTP server has
no on_connect hook, so only the h1_allow_half_closed call is added there.
Only the actix-web entry changes in Cargo.lock.
(cherry picked from commit d4b18b8)
The API server and the runner now call h1_allow_half_closed(false) directly, with a comment that tells why. The http_server module and its H1_ALLOW_HALF_CLOSED constant are removed. The test that a client disconnect drops a silent streaming body moves to a new test module in runner/main.rs and sets h1_allow_half_closed(false) on its own server. This matches the change on main.
The test started its own actix server with h1_allow_half_closed(false), so it tested actix, not the API server or the runner. It is removed to match the change on main.
Contributor
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Backport of #7963 (commit d4b18b8) to v0.334.2, for a customer whose kubernetes-runner memory grows until it is killed for being out of memory.
Problem
Each
GET /v0/pipelines/{name}/logsthat the client stops, on a pipeline that writes no new log lines, leaks one connection and about 50 KB in the API server and in the runner. Actix keeps a half-closed HTTP/1 connection open. A streaming response finds out that the client left only when a write fails, and a quiet logs stream does not write.Change
Cargo.toml,Cargo.lockactix-web4.11.0 -> 4.12.0 (first version whereHttpServerhash1_allow_half_closed). Only theactix-weblockfile entry changes.actix-httpis already 3.12.1.crates/pipeline-manager/src/api/main.rs,runner/main.rs.h1_allow_half_closed(false)on eachHttpServer, with a comment.At v0.334.2 the runner
HttpServerhas no.on_connect(Caller::record), so only the new call is added there.cargo update -p actix-web --precise 4.12.0also moved 19 unrelated edges (windows-sys, darling, socket2 for hyper-util and quinn), so the lockfile edit is limited to theactix-webentry.cargo metadata --lockedaccepts it with cargo 1.93.1 and 1.96.1.The logs endpoint at this version is the older one (no cursor protocol).
get_logsin the runner andhttp_streaming_logs_from_pipeline_by_namein the API server do not change. When the client closes, the API server drops its awc response to the runner, the runner drops the follower receiver, and the logs thread removes the closed follower on its idle cleanup tick.The three commits end in the same state as the change on main: no constant, no
http_servermodule, no test. Squash on merge.Base branch
v0.334.3is a copy ofv0.334.2(aced2c37). No version bump and no tag in this PR.