← 输入框|github.com|刷新
Skip to content

[python] Serialize Delta Spark runs across fixtures - #7863

Merged
Karakatiza666 merged 1 commit into
mainfrom
fred/delta-spark-global-build-lock
Nov 9, 2026
Merged

Karakatiza666 merged 1 commit into
mainfrom
fred/delta-spark-global-build-lock

Conversation

@mythical-fred-ops

Copy link
Copy Markdown
Contributor

Problem

ensure_delta_spark_fixture locks per fixture. xdist workers that build different fixtures still run Spark at the same time, and every run shares two caches that break with concurrent writers:

Cache Symptom
uv sdist cache (pyspark) Timeout (300s) when waiting for lock
Ivy cache (~/.ivy2.*) impossible to move part file, 0-byte jars, Premature end of file

Retries don't help: every contender retries on the same schedule.

Fix

  • run_delta_spark now holds a single host-wide flock (_fixture_build_lock("delta-spark")) around the subprocess. This covers fixture builds and direct callers such as test_delta_output_merge.
  • Lock order is always per-fixture, then global, so it cannot deadlock. The lock is released between retries.
  • Runs are rare (about 2 min each), so serializing them costs little and keeps cache reuse.

Testing

  • New test_builders_of_different_fixtures_do_not_overlap: two spawned processes build different fixtures with a stubbed Spark run, and the test asserts the run windows do not overlap.
    • Without the fix: fails (the windows overlap).
    • With the fix: all 5 tests in tests/unit/test_delta_fixture_build_lock.py pass.
  • pre-commit run --files <changed>: passes (ruff and ruff-format).

ensure_delta_spark_fixture locks each fixture, but xdist workers building
different fixtures still run Spark at once. Every run shares uv's package
cache and Spark's Ivy cache, which do not tolerate concurrent writers: uv
times out waiting on its sdist lock and Ivy leaves truncated jars. Retries
do not help because every contender retries on the same schedule.

Take one host-wide flock around the subprocess in run_delta_spark. This
covers fixture builds and direct callers. Runs are rare, so serializing
them is cheap and keeps cache reuse.

Signed-off-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com>
@gz
gz requested a review from swanandx November 6, 2026 07:46
@swanandx
swanandx added this pull request to the merge queue Nov 6, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Nov 6, 2026
@mythical-fred-oss

Copy link
Copy Markdown
Contributor

Verdict: infrastructure flake, not this PR's fault (high confidence). This PR only touches python/tests/unit/test_delta_fixture_build_lock.py and python/tests/utils.py; nothing in js-packages/web-console.

Evidence

Failed job Web Console End-to-End Tests / Web Console Integration Tests
Failure bun install --frozen-lockfile step, before any test ran
Error (log lines 406-407) error: TarballFailedToDownload downloading tarball apache-arrow@github:Karakatiza666/arrow-js#dist

The install couldn't fetch the git-hosted apache-arrow tarball dependency from GitHub — a transient network issue reaching a third-party ref, unrelated to anything in this PR.

Tracked in #7873.

Next step: re-queueing should pass; no action needed on this PR.

@swanandx
swanandx added this pull request to the merge queue Nov 9, 2026
@Karakatiza666

Karakatiza666 commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

Requeued, this will fail due to #7898

@Karakatiza666
Karakatiza666 removed this pull request from the merge queue due to a manual request Nov 9, 2026
@Karakatiza666
Karakatiza666 added this pull request to the merge queue Nov 9, 2026
@mythical-fred-oss

Copy link
Copy Markdown
Contributor

Verdict: pre-existing bug on main, not this PR's fault. High confidence.

The merge queue run's only failed job, Integration Tests / Runtime Integration Tests (gke-runners-arm64), failed on tests/workloads/test_kafka_avro.py::TestKafkaAvro::test_kafka_avro_variants (job, log lines 1090-1160):

RuntimeError: Unable to START the pipeline:
error generating Avro schema for the SQL relation 708ef_test_kafka_avro_1_view:
'708ef_test_kafka_avro_1_view' is not a valid Avro name: each dot-separated
segment must start with a letter or underscore ...

This is the exact bug already tracked in #7898: unique_pipeline_name (python/feldera/testutils.py:244) prefixes names with the first 5 hex chars of GITHUB_SHA, which starts with a digit ~62% of the time, breaking Avro name validation for any view exported as Avro. This run's merge commit started with a digit (708ef...); unrelated to this PR's diff (python/tests/unit/test_delta_fixture_build_lock.py, python/tests/utils.py).

Added this run's evidence as a comment on #7898 rather than opening a duplicate. Re-queueing should pass once the commit SHA doesn't start with a digit, or once #7898 is fixed.

Merged via the queue into main with commit c9d8c8a Nov 9, 2026
3 checks passed
@Karakatiza666
Karakatiza666 deleted the fred/delta-spark-global-build-lock branch November 9, 2026 18:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants