Repository navigation
[connectors] Fix the flaky DynamoDB retry tests (#7828) - #7838
Merged
Merged
Conversation
…rns_unhealthy_after_the_threshold The test asserted that reports() was still empty right after the chunk entered its retry wait, racing a 500ms real-wall-clock threshold. On a busy CI runner the test thread can lose the scheduler for that long before it gets to the assertion, so the health report had already landed. Raise the threshold to 3s to give the check a realistic margin; the test's own 10s bound on the opposite wait still leaves plenty of room for the report to show up. Seen twice in the merge queue for #7736 (runs 41750935279 and 41850195763), tracked in #7828. Signed-off-by: mythical-fred-oss[bot] <319348225+mythical-fred-oss[bot]@users.noreply.github.com>
…d one `MockServer::start()` hands out servers from a pool and recycles them when a test drops its server. A chunk writer that a failed test leaves behind on the shared runtime can send one more request, which then lands on the recycled server of whichever test picked it up next, and that test counts it as its own. In CI, `the_retry_budget_is_exact` saw two requests instead of one right after `a_long_retry_loop_turns_unhealthy_after_the_threshold` failed. `MockServer::builder().start()` returns a server that is not pooled, so a stray request only ever reaches the server of the test that made it. Reproduced with a panic injected after the chunk entered its retry and the released task delayed by 100 ms: 5 of 10 runs broke another retry test with the pooled server, 0 of 10 with this change. Refs #7828.
| // its retry wait, not that this thread resumes promptly enough | ||
| // to see `reports()` still empty against a tight threshold on a | ||
| // busy CI runner (see #7828). | ||
| RetryPolicy::with_delay(None, delay).unhealthy_after(Duration::from_secs(3)), |
Contributor
There was a problem hiding this comment.
pitfall (medium): The 3 s threshold makes a flake much less likely, but the reports().is_empty() assertion below still races a wall clock. You can make it deterministic: the timer starts at the first retry, which happens after write_batch is called. So take an Instant just before write_batch and only assert when less time than the threshold has passed:
let start = std::time::Instant::now();
// ... write_batch / gate.entered() ...
if start.elapsed() < Duration::from_secs(3) {
assert!(harness.publisher.reports().is_empty());
}The threshold check stays correct, and a stalled runner can no longer cause a false failure.
Contributor
|
What I ran (debug,
Otherwise LGTM. This PR only changes tests, so the docs, type-coverage and breaking-change gates don't apply. |
The timer that reports a long retry loop unhealthy starts at the chunk's first retry, which is after the test calls `write_batch`. So an instant taken before that call bounds the timer: while less than the threshold has passed since then, no report can exist yet, and the assertion holds by construction. A runner that stalls past the threshold skips the check instead of failing it, so the test no longer races a wall clock. Refs #7828.
gz
approved these changes
Nov 4, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #7828, on top of #7736.
Two commits, one per failure in the issue:
a_long_retry_loop_turns_unhealthy_after_the_thresholdasserts no reportunhealthy_aftertimer is armed before the gate fires, so a slow runner can pass it before the assertion runs0df319db0)the_retry_budget_is_exactsees 2 requests instead of 1Gatebefore itsMockServer. The chunk task on the shared runtime wakes and sends one more request, which lands on the recycled pooled server that the second test picked up 80 ms laterMockServer::builder().start(), which is not pooledReproduction of the second failure, with a panic injected after the chunk enters its retry and the released task delayed by 100 ms: