← 输入框|github.com|刷新
Skip to content

Close a connection when the client closes it - #7963

Merged
xiasongh merged 3 commits into
mainfrom
song/debug-customer-issue-61bc4c
Nov 13, 2026
Merged

xiasongh merged 3 commits into
mainfrom
song/debug-customer-issue-61bc4c

Conversation

@xiasongh

@xiasongh xiasongh commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Issue

A client that disconnects from a streaming response that writes nothing leaks the connection and the stream's state. Actix keeps the half-closed connection open until a write fails, and nothing is written.

  • API server and runner: GET /v0/pipelines/{name}/logs on a pipeline that writes no new lines leaks a connection in both, and a log follower in the runner. Memory grows with each such request until the runner is OOM-killed.
  • Pipeline server: the same happens on /coordination/status, /coordination/step/status, /coordination/checkpoint/status and similar while the value does not change.

Fix

All three servers set h1_allow_half_closed(false), so they close the connection when they read EOF from the client. HttpServer exposes this setting from actix-web 4.12, so actix-web goes from 4.11.0 to 4.12.0.

A disconnect now cancels the handler, so Controller::rebalance requests the step from the command callback instead of after it awaits the reply.

@mythical-fred-oss

Copy link
Copy Markdown
Contributor

What I ran

Check Result
cargo test --locked -p pipeline-manager --lib http_server ✅ passes
Same test with H1_ALLOW_HALF_CLOSED = true ❌ fails after 5 s, so the test catches the regression
Ran the test 30 times in a row ✅ 30/30, finishes in ~10 ms, no sign of flakiness
pre-commit run --files <changed> (clippy, fmt, machete, crypto deps, OpenAPI) ✅ all pass
actix-web 4.11 → 4.12 bump minor bump, MIT/Apache license; socket2 0.6.5 was already in the lockfile

Gaps (medium)

  • missing: the pipeline's own server (crates/connectors/src/server.rs, HttpServer::new around L1153) still uses the default h1_allow_half_closed. Its egress and ad hoc streams can be quiet for a long time, e.g. /egress on a view that doesn't change. So a client that walks away there likely still leaves a connection and output handle inside the pipeline process. Please apply the same setting there, or say why it isn't needed.
  • missing: the test covers the actix setting alone, not the api-server → runner chain from the description (API server drops the upstream request → runner drops the follower → logs thread removes it). Nothing checks that the follower count goes back down after an aborted /logs. A runner-level test that opens /logs, drops the client, and checks that the follower is removed would protect this against future refactors.

Otherwise LGTM: the fix is small and well explained, and the commit message is good.

@xiasongh
xiasongh force-pushed the song/debug-customer-issue-61bc4c branch from b524436 to 2ac9d6a Compare November 13, 2026 01:21
@xiasongh
xiasongh added this pull request to stack #7967 November 13, 2026 01:32
@xiasongh xiasongh changed the title [pipeline-manager] Close a connection when the client closes it [pipeline-manager] [adapters] Close a connection when the client closes it Nov 13, 2026
@xiasongh xiasongh changed the title [pipeline-manager] [adapters] Close a connection when the client closes it Close a connection when the client closes it Nov 13, 2026
It is the first release whose `HttpServer` exposes `h1_allow_half_closed`.
A client that stops a /logs request on a pipeline that writes no new
lines leaked a connection in the API server and the runner, and a log
follower in the runner. Actix kept the half-closed connection open until
a write failed, and nothing was written.

The API server and the runner now close the connection when they read EOF
from the client. This needs actix-web 4.12.
A client that disconnects from a streaming response that writes nothing,
such as /coordination/status while the status does not change, leaked the
connection and the stream's state. Actix kept the half-closed connection
open until a write failed.

The pipeline's HTTP server now closes the connection when it reads EOF
from the client. Because a disconnect now cancels the handler,
Controller::rebalance requests the step from the command callback instead
of after it awaits the reply.
@xiasongh
xiasongh force-pushed the song/debug-customer-issue-61bc4c branch from 5eb1e44 to 8c985bd Compare November 13, 2026 01:39
self.inner
.send_command(Command::Rebalance(Box::new(move |result| {
if result.is_ok() {
inner.request_step();

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do we need to make this change?

@blp blp left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is fascinating, thanks.

I wonder whether we use any half-closed connections in practice intentionally (I simply don't know).

@xiasongh
xiasongh added this pull request to the merge queue Nov 13, 2026
Merged via the queue into main with commit 969d071 Nov 13, 2026
1 check passed
@xiasongh
xiasongh deleted the song/debug-customer-issue-61bc4c branch November 13, 2026 05:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants