Repository navigation
[python] Serialize Delta Spark runs across fixtures - #7863
Conversation
ensure_delta_spark_fixture locks each fixture, but xdist workers building different fixtures still run Spark at once. Every run shares uv's package cache and Spark's Ivy cache, which do not tolerate concurrent writers: uv times out waiting on its sdist lock and Ivy leaves truncated jars. Retries do not help because every contender retries on the same schedule. Take one host-wide flock around the subprocess in run_delta_spark. This covers fixture builds and direct callers. Runs are rare, so serializing them is cheap and keeps cache reuse. Signed-off-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com>
|
Verdict: infrastructure flake, not this PR's fault (high confidence). This PR only touches Evidence
The install couldn't fetch the git-hosted Tracked in #7873. Next step: re-queueing should pass; no action needed on this PR. |
|
Requeued, this will fail due to #7898 |
|
Verdict: pre-existing bug on main, not this PR's fault. High confidence. The merge queue run's only failed job, This is the exact bug already tracked in #7898: Added this run's evidence as a comment on #7898 rather than opening a duplicate. Re-queueing should pass once the commit SHA doesn't start with a digit, or once #7898 is fixed. |
Problem
ensure_delta_spark_fixturelocks per fixture. xdist workers that build different fixtures still run Spark at the same time, and every run shares two caches that break with concurrent writers:pyspark)Timeout (300s) when waiting for lock~/.ivy2.*)impossible to move part file, 0-byte jars,Premature end of fileRetries don't help: every contender retries on the same schedule.
Fix
run_delta_sparknow holds a single host-wideflock(_fixture_build_lock("delta-spark")) around the subprocess. This covers fixture builds and direct callers such astest_delta_output_merge.Testing
test_builders_of_different_fixtures_do_not_overlap: two spawned processes build different fixtures with a stubbed Spark run, and the test asserts the run windows do not overlap.tests/unit/test_delta_fixture_build_lock.pypass.pre-commit run --files <changed>: passes (ruff and ruff-format).