Repository navigation
Conversation
An exchange client's task ran forever, resending unacknowledged messages to its receiver's port. In tests, a later host on that port received them; since receive-task panics fail the host, stale messages left by the exchange wire tests failed sharded-accumulator tests with UnknownExchange(0). Dropping a client now lets its task finish delivering the messages already queued, because a host can finish while its peers still wait for its last round. The task stops once the receiver has acknowledged every message, or once a connection is refused, which means that the receiver's listener is gone. Fixes #7800. Signed-off-by: Leonid Ryzhyk <ryzhyk@gmail.com>
|
I think this is a subset of the first commit in #7814 , which is currently queued. |
| Ok(stream) => break stream, | ||
| Err(error) => error, | ||
| }; | ||
| if dropped.is_cancelled() && error.kind() == std::io::ErrorKind::ConnectionRefused { |
There was a problem hiding this comment.
bug (medium): the race is smaller now, but it is still there. After the connection breaks, run reconnects right away. If a new listener has already bound the same port by then, connect succeeds and the stale message gets delivered. That is the UnknownExchange failure from #7800 again. I reproduced it with this test (it fails on this branch):
let (client, _task) = ExchangeClient::new_with_task(MessageType::Streaming, addr, &(0..1), cap, None);
let stream = accept_client(&listener).await;
client.send(Arc::new("test".into()), 7, 0, 0, vec![FBuf::new()]);
drop(client);
drop(listener);
let listener2 = AsyncTcpListener::bind(addr).await.unwrap();
drop(stream);
// accept_client(&listener2) gets a header with exchange_id 7host_configs takes its ports from 127.0.0.1:0, so this needs the kernel to hand out a port again. That is rare, but it is the kind of rare that makes tests flaky. To actually close the race, the receiver needs some way to tell this sender apart from the right one, e.g. a per-run/session nonce in the magic handshake, which the server would check and reject.
pitfall (medium): the task can still leak in production. Only ConnectionRefused stops the task. When a peer pod or VM disappears, connect usually fails with a connect or handshake timeout, EHOSTUNREACH/ENETUNREACH, or InvalidData (TLS). In those cases a dropped client keeps retrying forever and holds its queued buffers. Consider an upper bound on retries, or a deadline, once dropped is cancelled.
|
What I ran (debug,
Gates: tests were added and there are no Missing (medium): these cases have no test:
The fix is an improvement and fine to merge as is, but I'd keep #7800 open, or file a follow-up for a per-session handshake check. |
perfect! |
This fixes a recent test suite regression I introduced in the sharded accumulator.
An exchange client's task ran forever, resending unacknowledged messages to its receiver's port. In tests, a later host on that port received them; since receive-task panics fail the host, stale messages left by the exchange wire tests failed sharded-accumulator tests with UnknownExchange(0).
Dropping a client now lets its task finish delivering the messages already queued, because a host can finish while its peers still wait for its last round. The task stops once the receiver has acknowledged every message, or once a connection is refused, which means that the receiver's listener is gone.
Linked Issue
Fixes #7800.
Describe Manual Test Plan
Checklist
Breaking Changes?
Mark if you think the answer is yes for any of these components:
Describe Incompatible Changes