Repository navigation
CI AutoFormat fails when CI is under load. #6272
Description
The current CI auto-format implementation waits for all jobs to be finished before proceeding, and that wait time is 1 hour.
In the case where the auto-format job gets scheduled before the rest of the other jobs, it will start it's countdown immediately, but there's a chance for the other jobs to still be in queue because some other CI jobs are running for a pushed commit/other PR, which might take more than 1 hour.
It looks somewhat like this:
gantt
title CI Timeline
dateformat HH:mm
axisFormat %H:%M
PR Auto Format: 00:15, 1h
Task 1: 00:00, 45m
Task 2: 00:45, 45m
Fails if still waiting: milestone, m1, 01:15, 0m
There are several approaches that I think could work to solve this:
- increase waiting timeout (or remove the timeout).
- Make sure that the PR auto format is triggered before any CI jobs.
- Make the auto-formag opt-in format by triggering it via a comment on the PR, just like we have
takeassigning people to issues (https://github.com/RustPython/RustPython/blob/cc2e84b9fc0af690f3281cd4b02e219b4752bed7/.github/workflows/comment-commands.yml)
I'm leaning towards having the ability to trigger it on-demand. And if not, then move it to the start of the CI as having a job just waiting for 99% of it's time is a waste of a runner that could be used to speed up the CI (at the time writing this we have 6 concurrent auto-pr format running, and 16 CI jobs in queue)
Activity
[-]CI AutoFormat fails when CI is under moderate load.[/-][+]CI AutoFormat fails when CI is under load.[/+]youknowone commentedon Nov 15, 2025
I prefer to run format task first. ref: #6204
Checking it before other workflow will be very cheap. because it is just one rustfmt run, which we already did by
cargo fmt --check. I don't see much value to maketakecommand.Currently the CI is under irregular pressure. We usually don't have that many waiting jobs.
ShaharNaveh commentedon Nov 15, 2025
Sure, will open a PR for it
That's true, but it emphasize the issue even more.
ShaharNaveh commentedon Nov 15, 2025
take
ShaharNaveh commentedon Dec 9, 2025
Closed by #6295