← 输入框|github.com|刷新
Skip to content

[connectors] Run the Delta CDC suspend case on aarch64 again - #7969

Merged
ryzhyk merged 1 commit into
mainfrom
issue-4176
Nov 13, 2026
Merged

ryzhyk merged 1 commit into
mainfrom
issue-4176

Conversation

@ryzhyk

@ryzhyk ryzhyk commented Nov 13, 2026

Copy link
Copy Markdown
Contributor

cdc_file_suspend_test has been skipped on aarch64 since June 2025, when it failed intermittently right after the arm64 adapter tests moved to self-hosted runners that ran them with no concurrency limit; the case then waited 30 s per chunk. CI now caps the number of concurrent tests, and the case waits up to 600 s per chunk.

Linked Issue

Fixes #4176.

Describe Manual Test Plan

Checklist

  • Unit tests added/updated
  • Integration tests added/updated
  • Documentation updated
  • Changelog updated

Breaking Changes?

Mark if you think the answer is yes for any of these components:

Describe Incompatible Changes

`cdc_file_suspend_test` has been skipped on aarch64 since June 2025, when it
failed intermittently right after the arm64 adapter tests moved to
self-hosted runners that ran them with no concurrency limit; the case then
waited 30 s per chunk. CI now caps the number of concurrent tests, and the
case waits up to 600 s per chunk.

Fixes #4176.

Signed-off-by: Leonid Ryzhyk <ryzhyk@gmail.com>
@ryzhyk
ryzhyk requested a review from swanandx November 13, 2026 03:10
@ryzhyk ryzhyk added the connectors Issues related to the adapters/connectors crate label Nov 13, 2026
ignore_if(target_arch = "aarch64", "flaky on aarch64"),
requires(Storage, Checkpoints, SuspendResume)
)]
#[engine_test(requires(Storage, Checkpoints, SuspendResume))]

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pitfall (medium): The 600 s argument only covers wait_for_records_materialized. This case also runs the suspend path in test_cdc, which still waits on a hardcoded timeout(Duration::from_secs(100), receiver.recv()), once per chunk (11 suspends). That is the only wait that only this case reaches, so it is the one an arm64 runner under load would hit first. The June 2025 log has expired, so we can't tell which wait failed. Use Duration::from_millis(default_timeout_ms()) there too, so that the reasoning in the commit message holds for every wait in the case.

@mythical-fred-oss

Copy link
Copy Markdown
Contributor

What I ran (x86_64, debug; no arm64 host was available):

Check Result
cargo test --locked -p static-connectors --test connector_suites --no-run builds
connector_suites --exact …cdc_file_suspend_test pass, 102 s
6 copies at once, pinned to 4 cores (taskset -c 0-3) to simulate a starved runner 6/6 pass, 50–160 s each

Checked the claims: default_timeout_ms() is 600 s, and test-adapters.yml caps RUST_TEST_THREADS at CI_RUNNER_CORES.

Flaky-test gate, medium: this passes provided the 100 s suspend timeout also moves to the default wait (see the inline comment). The commit message gives a plausible cause but no evidence for it, and the original failure log has expired. This PR's own aarch64 adapter job is the only real evidence that the case is stable there. Please re-run that job a few times before merging. If the case flakes again, reopen #4176 instead of skipping it again without an issue.

@ryzhyk
ryzhyk added this pull request to the merge queue Nov 13, 2026
Merged via the queue into main with commit d80f681 Nov 13, 2026
3 of 4 checks passed
@ryzhyk
ryzhyk deleted the issue-4176 branch November 13, 2026 08:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

connectors Issues related to the adapters/connectors crate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

non deterministic failures for delta_table_cdc_file_suspend_test

2 participants