← 输入框|github.com|刷新
Skip to content

[adapters] Close a connection when the client closes it - #7966

Merged
xiasongh merged 1 commit into
song/debug-customer-issue-61bc4cfrom
song/pipeline-server-half-closed
Nov 13, 2026
Merged

xiasongh merged 1 commit into
song/debug-customer-issue-61bc4cfrom
song/pipeline-server-half-closed

Conversation

@xiasongh

Copy link
Copy Markdown
Contributor

Stacked on #7963, which bumps actix-web to 4.12.

Issue

A client that disconnects from a streaming response that writes nothing (for example /coordination/status, /coordination/step/status, /coordination/checkpoint/status while the value does not change) leaks the connection and the stream's state. Actix keeps the half-closed connection open until a write fails.

Fix

The pipeline's HTTP server sets h1_allow_half_closed(false), so it closes the connection when it reads EOF from the client. A disconnect now cancels the handler, so Controller::rebalance requests the step from the command callback instead of after it awaits the reply.

A client that disconnects from a streaming response that writes nothing,
such as /coordination/status while the status does not change, leaked the
connection and the stream's state. Actix kept the half-closed connection
open until a write failed.

The pipeline's HTTP server now closes the connection when it reads EOF
from the client. Because a disconnect now cancels the handler,
Controller::rebalance requests the step from the command callback instead
of after it awaits the reply.
@xiasongh
xiasongh added this pull request to stack #7967 November 13, 2026 01:32
@xiasongh
xiasongh merged commit 5eb1e44 into main Nov 13, 2026
3 checks passed
@xiasongh
xiasongh deleted the song/pipeline-server-half-closed branch November 13, 2026 01:35
// Close the connection when the client closes it. Otherwise a streaming response that
// writes nothing, such as `/coordination/status` while the status does not change,
// leaks its connection until a write fails.
.h1_allow_half_closed(false)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pitfall (medium): This applies to every handler, not only streaming responses. I tested it with a real HttpServer. A client sent POST with a full body and then called shutdown(SHUT_WR), which HTTP/1.1 allows. It received 0 bytes, and the handler future was dropped at its next .await. For the pipeline, that means:

  • /input_endpoint while paused: the request is cancelled in status_watch.changed(), so its data is never ingested, and no error is logged.
  • /approve, clock_advance, and similar handlers are cancelled partway.

The manager's clients (reqwest/awc) don't half-close, so this is probably fine in practice. Please note the trade-off in this comment. Also please audit for other send-command → await → side-effect handlers like the old rebalance. I found none besides rebalance, but new ones will need the same pattern from now on.

@mythical-fred-oss

Copy link
Copy Markdown
Contributor

What I ran: cargo check --locked -p dbsp_connectors ✅ and rustfmt --check on both files ✅. I also wrote scratch tests that run a real HttpServer (results below).

Scratch test h1_allow_half_closed(true) (false) (this PR)
Client half-closes on a silent stream stream kept alive (leak reproduced) stream dropped, connection closed ✅
Client half-closes after a full POST to a slow handler — 0-byte response, handler cancelled (see inline comment)

missing (high) – feldera-pr-checks gate #2 (unit tests) fails: This changes behaviour and adds no test. The actix_test server (test_server_config()) cannot set h1_allow_half_closed, so no existing test runs with the new setting. Please move the HttpServer connection settings into a shared helper. Then add a test that opens a /coordination/status stream, half-closes the socket, and asserts that the server closes the connection (read returns EOF within a bounded timeout). Sketch:

let mut sock = TcpStream::connect(addr).await?;
sock.write_all(b"GET /coordination/status HTTP/1.1\r\nHost: x\r\n\r\n").await?;
// read the response headers and the first status chunk, then:
sock.shutdown().await?;
timeout(Duration::from_secs(10), async { while sock.read(&mut buf).await? > 0 {} }).await.expect("server closed the connection");

missing (medium): Nothing tests the rebalance change, which is that the step is still requested when the caller is dropped before the reply arrives. A controller-level test would cover it: send rebalance(), drop the future, then assert that a step happens.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant