← 输入框|github.com|刷新
Skip to content

[dbsp] End an exchange client's task after the client is dropped - #7842

Closed
ryzhyk wants to merge 1 commit into
mainfrom
issue-7800
Closed

ryzhyk wants to merge 1 commit into
mainfrom
issue-7800

Conversation

@ryzhyk

@ryzhyk ryzhyk commented Nov 4, 2026

Copy link
Copy Markdown
Contributor

This fixes a recent test suite regression I introduced in the sharded accumulator.

An exchange client's task ran forever, resending unacknowledged messages to its receiver's port. In tests, a later host on that port received them; since receive-task panics fail the host, stale messages left by the exchange wire tests failed sharded-accumulator tests with UnknownExchange(0).

Dropping a client now lets its task finish delivering the messages already queued, because a host can finish while its peers still wait for its last round. The task stops once the receiver has acknowledged every message, or once a connection is refused, which means that the receiver's listener is gone.

Linked Issue

Fixes #7800.

Describe Manual Test Plan

Checklist

  • Unit tests added/updated
  • Integration tests added/updated
  • Documentation updated
  • Changelog updated

Breaking Changes?

Mark if you think the answer is yes for any of these components:

Describe Incompatible Changes

An exchange client's task ran forever, resending unacknowledged messages
to its receiver's port.  In tests, a later host on that port received
them; since receive-task panics fail the host, stale messages left by the
exchange wire tests failed sharded-accumulator tests with
UnknownExchange(0).

Dropping a client now lets its task finish delivering the messages
already queued, because a host can finish while its peers still wait for
its last round.  The task stops once the receiver has acknowledged every
message, or once a connection is refused, which means that the
receiver's listener is gone.

Fixes #7800.

Signed-off-by: Leonid Ryzhyk <ryzhyk@gmail.com>
@ryzhyk
ryzhyk requested a review from blp November 4, 2026 21:32
@ryzhyk ryzhyk added DBSP core Related to the core DBSP library multihost Related to multihost or distributed pipelines labels Nov 4, 2026
@blp

blp commented Nov 4, 2026

Copy link
Copy Markdown
Member

I think this is a subset of the first commit in #7814 , which is currently queued.

Ok(stream) => break stream,
Err(error) => error,
};
if dropped.is_cancelled() && error.kind() == std::io::ErrorKind::ConnectionRefused {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bug (medium): the race is smaller now, but it is still there. After the connection breaks, run reconnects right away. If a new listener has already bound the same port by then, connect succeeds and the stale message gets delivered. That is the UnknownExchange failure from #7800 again. I reproduced it with this test (it fails on this branch):

let (client, _task) = ExchangeClient::new_with_task(MessageType::Streaming, addr, &(0..1), cap, None);
let stream = accept_client(&listener).await;
client.send(Arc::new("test".into()), 7, 0, 0, vec![FBuf::new()]);
drop(client);
drop(listener);
let listener2 = AsyncTcpListener::bind(addr).await.unwrap();
drop(stream);
// accept_client(&listener2) gets a header with exchange_id 7

host_configs takes its ports from 127.0.0.1:0, so this needs the kernel to hand out a port again. That is rare, but it is the kind of rare that makes tests flaky. To actually close the race, the receiver needs some way to tell this sender apart from the right one, e.g. a per-run/session nonce in the magic handshake, which the server would check and reject.

pitfall (medium): the task can still leak in production. Only ConnectionRefused stops the task. When a peer pod or VM disappears, connect usually fails with a connect or handshake timeout, EHOSTUNREACH/ENETUNREACH, or InvalidData (TLS). In those cases a dropped client keeps retrying forever and holds its queued buffers. Consider an upper bound on retries, or a deadline, once dropped is cancelled.

@mythical-fred-oss

Copy link
Copy Markdown
Contributor

What I ran (debug, -p dbsp --lib):

Run Result
dropped_client_* (the 2 new tests) ✅ pass
exchange filter (33 tests), run 3 times ✅ all pass
accumulat filter (49 tests) ✅ pass
My extra test: drop an idle client (nothing queued) ✅ the task stops
My extra test: drop the client, then rebind the listener's port ❌ the stale message is delivered (see inline comment)

Gates: tests were added and there are no unsafe blocks, dependency changes or user-visible docs changes, so the gates pass. The commit message is good.

Missing (medium): these cases have no test:

The fix is an improvement and fine to merge as is, but I'd keep #7800 open, or file a follow-up for a per-session handshake check.

@ryzhyk

ryzhyk commented Nov 4, 2026

Copy link
Copy Markdown
Contributor Author

I think this is a subset of the first commit in #7814 , which is currently queued.

perfect!

@ryzhyk ryzhyk closed this Nov 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

DBSP core Related to the core DBSP library multihost Related to multihost or distributed pipelines

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[dbsp] Flaky: sharded_accumulator::tests::take_received panics with UnknownExchange on multihost harness

2 participants