Repository navigation
ci: shard CodSpeed benchmarks across jobs instead of xdist processes - #669
Conversation
Follow-up to commit-check#668. commit-check#668 parallelized the benchmark job with pytest-xdist (-n auto), which cut it from ~54 min to ~10 min but left the CodSpeed check red with a false 11.1% regression: running a syscall-bound benchmark in an xdist worker shifts its simulated number even though no code changed. Those false regressions will recur on future PRs and erode the gate, so trade in-process parallelism for sharding across CI jobs. Each shard runs single-process, so benchmarks are measured in the same context as a serial run (stable numbers); CodSpeed aggregates the shards. This is the approach the CodSpeed docs list first. - codspeed.yml: strategy.matrix.shard [1..5]; run pytest with --splits/--group; fail-fast: false. - pyproject.toml: [test] pytest-xdist -> pytest-split; uv.lock relocked. Uses pytest-split, not the docs' pytest-test-groups: under --codspeed on pytest 9 the latter does not split (every shard runs the full set). pytest-split partitions into 5 disjoint groups (133/133/133/133/129=661); a shard runs green locally.
|
No actionable comments were generated in the recent review. Walkthrough
Merge Risk:
|
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #669 +/- ##
=========================================
Coverage 100.00% 100.00%
=========================================
Files 14 14
Lines 2495 2495
Branches 484 484
=========================================
Hits 2495 2495 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
Moving this to draft — the sharding works, but I can't validate CodSpeed's shard aggregation from a fork PR, and I don't want it merged until that's confirmed in-repo. Details: What works (verified on this PR's CI run):
The problem: CodSpeed posted no aggregated report on this PR — no What it is not: not an auth/OIDC gap. I checked a repo that shards CodSpeed successfully ( Best read: the combination that fails here is fork PR + sharding. #668 (fork, single job) reported fine; in-repo sharded setups report fine; but CodSpeed doesn't seem to correlate several tokenless uploads from a fork PR's workflow run into one report. The CodSpeed docs say shards aggregate when "split within a single CI workflow" but don't document the fork case, and I couldn't find a public fork-PR sharded example that aggregated. Why this matters before merging: the Suggested next step: if you're open to the sharded approach, the decisive test is pushing this Happy to carry whichever direction you prefer. |
Merging this PR will improve performance by 12.47%3 warnings
Performance Changes
Tip Curious why performance improved? Comment Comparing Footnotes
|
|
Correction — I was wrong above, and I'm un-drafting. CodSpeed's aggregated report did post; it was just slow for the sharded run (~1h+ after the shards finished, vs minutes for a single job). Aggregation across the 5 shards worked fine on this fork PR — the five different run hashes I flagged were a red herring (each upload gets its own hash; CodSpeed still merges them by commit into one report). My earlier "fork + sharding doesn't aggregate" conclusion was premature impatience on my part. Apologies for the noise. The result is exactly what this PR was for —
The one changed benchmark is the same So the full picture is now confirmed end-to-end: 5 disjoint shards (133/133/133/133/129 = 661) via One real caveat still stands: the matrix turns the single Ready for review whenever you'd like to take a look. |



Follow-up to #668.
Why this follow-up, rather than leaving #668 as-is. #668 got the speedup (benchmark job ~54 min → ~10 min) but shipped it with the
CodSpeed Performance Analysischeck red — "Merging this PR will degrade performance by 11.1%" ontest_no_terminal_output_produced. That was a false positive (#668 changed nocommit_check/code; the benchmark iscapsys/stdout-bound, i.e. one of the "136 benchmarks dominated by system calls" CodSpeed cannot simulate consistently, and running it in an xdist worker process shifted its simulated number). The problem is that it isn't a one-off: with in-jobpytest-xdist -n auto, those syscall-dominated benchmarks will keep shifting across PRs, so the benchmark gate will keep throwing false regressions. A gate that cries wolf gets ignored — and then a real regression slips through unnoticed. This PR keeps #668's parallelism and speedup while removing that failure mode.The fix: shard across CI jobs instead of across processes in one job. Each shard runs single-process, so every benchmark is measured in the same context as the old serial run — the simulated numbers are stable — and CodSpeed aggregates the shards into one report. This is the approach the CodSpeed docs list first (
-n autois offered only as a fallback "if you cannot split across CI jobs").Why pytest-split and not the pytest-test-groups in the CodSpeed docs. On this project's pytest (9.0.3),
pytest-test-groupsdoes not split when--codspeedis active — pytest-codspeed and pytest-test-groups both wrappytest_collection_modifyitems, and the net result is that every shard collects the full benchmark set (which is exactly the "same benchmark run multiple times" case the docs warn against).pytest-splitpartitions the--codspeedselection correctly: 5 disjoint groups of 133/133/133/133/129 = 661 benchmarks, verified locally, and a shard runs green.Change
.github/workflows/codspeed.yml:strategy.matrix.shard: [1..5];run: pytest tests/ --codspeed --splits 5 --group ${{ matrix.shard }}. (fail-fast: falseso one shard's result doesn't cancel the others.)pyproject.toml:[test]extrapytest-xdist→pytest-split.uv.lockregenerated.Note on check names. The matrix turns the single
Run benchmarkscheck intoRun benchmarks (1..5). IfRun benchmarksis a required status check in branch protection, the required-check name will need updating to match.Tuning. 5 shards ≈ 11 min/shard here; bump the matrix list for more parallelism (each shard has ~1 min fixed setup).
Summary by CodeRabbit