← 输入框|github.com|刷新
Skip to content

[connectors] NATS: wait on the server for a library-recreated consumer. - #7824

Merged
bestouff merged 1 commit into
mainfrom
nats-consumer-count-race
Nov 4, 2026
Merged

bestouff merged 1 commit into
mainfrom
nats-consumer-count-race

Conversation

@blp

@blp blp commented Nov 3, 2026

Copy link
Copy Markdown
Member

Fixes the flaky library_recreated_consumer_is_deleted_on_pause test.

Cause

The test deletes the consumer itself, then uses WaitForConsumerCount to wait for async-nats to recreate it. WaitForConsumerCount polls the connector's own created - deleted count:

Step Server consumers Connector count
CreatePipeline, Extend 1 1
DeleteConsumers (by the test) 0 1 (the connector does not see this delete)
WaitForConsumerCount{1} poll 0 1, so the poll returns at once
single server check 0 or 1 (race with the library's recreate)

On a loaded CI runner, the server check came first:

Connector's own count reached 1, but the server disagrees: found 0 consumers on stream 'str': []

Fix

Add WaitForServerConsumerCount, which polls the server's consumer list, and use it for this wait. WaitForConsumerCount stays the same where the connector makes every create and delete.

Testing

  • Delayed the library's recreate by 2 s in the test proxy (temporary change): the old wait fails with the error above, and the new wait passes.
  • NATS engine suite: 70 cases pass.

Closes #7820

library_recreated_consumer_is_deleted_on_pause deletes the consumer from
the test, then uses WaitForConsumerCount to wait for the client library
to recreate it. WaitForConsumerCount polls the connector's own
created - deleted count. The connector does not count a delete from
outside or a recreate by the library, so that count stays at 1 and the
wait returns at once. The single server check after it then raced the
recreate, and lost on a loaded CI runner:

    Connector's own count reached 1, but the server disagrees: found 0
    consumers on stream 'str': []

Add WaitForServerConsumerCount, which polls the server's consumer list,
and use it for this wait.

Tested by delaying the library's recreate by 2 s in the test proxy: the
old wait fails with the error above and the new wait passes. The NATS
engine suite (70 cases) passes.

Closes #7820

Signed-off-by: Ben Pfaff <blp@feldera.com>
@blp
blp requested a review from bestouff November 3, 2026 23:51
@blp blp self-assigned this Nov 3, 2026
@blp blp added bug Something isn't working connectors Issues related to the adapters/connectors crate CI/CD QA Testing and quality assurance rust Pull requests that update Rust code labels Nov 3, 2026
@mythical-fred-oss

Copy link
Copy Markdown
Contributor

LGTM. This is a test-only fix: the new wait polls the server's consumer list, which is the right source of truth after an external delete. The 50 ms poll has a deadline, so it is justified and does not add flakiness.

Ran Result
cargo test -p static-connectors --test connector_suites -- library_recreated_consumer pass
Same test 30× with 8 CPU-burning yes processes running 30/30 pass
connector_suites -- transport::nats:: 70/70 pass
rustfmt --check --edition 2024 on the changed files clean

@bestouff
bestouff added this pull request to the merge queue Nov 4, 2026
Merged via the queue into main with commit 490e56f Nov 4, 2026
3 checks passed
@bestouff
bestouff deleted the nats-consumer-count-race branch November 4, 2026 14:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working CI/CD connectors Issues related to the adapters/connectors crate QA Testing and quality assurance rust Pull requests that update Rust code

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Flaky: nats::hardening_tests::library_recreated_consumer_is_deleted_on_pause — consumer count race

2 participants