← 输入框|github.com|刷新
Skip to content

[python] Time out and retry hung Delta Spark subprocesses - #7726

Merged
bestouff merged 2 commits into
mainfrom
fred/delta-spark-subprocess-timeout
Nov 13, 2026
Merged

bestouff merged 2 commits into
mainfrom
fred/delta-spark-subprocess-timeout

Conversation

@mythical-fred-ops

Copy link
Copy Markdown
Contributor

run_delta_spark (python/tests/utils.py) runs its Spark subprocess with no timeout. Both retry loops (there and in _build_delta_spark_fixture) catch only CalledProcessError. So when Spark hangs in Ivy dependency resolution instead of exiting, nothing retries it, and the test blocks until pytest-timeout kills it.

Changes

Where Change
run_delta_spark new timeout_s=300 parameter, passed to subprocess.run(timeout=...)
both retry loops retry on TimeoutExpired as well as CalledProcessError, and log the exception
tests/unit/test_delta_fixture_build_lock.py new test_hung_builder_is_timed_out_and_retried

300s is about twice a healthy cold build. Three attempts plus backoff come to about 915s, which still fits under the 1500s pytest timeout.

Testing

  • uv run pytest tests/unit/test_delta_fixture_build_lock.py: 5 passed.
  • With the utils.py change reverted, the new test fails ("the builder subprocess must be run with a timeout").
  • pre-commit run --files <changed files>: ruff and ruff-format pass.

@gz
gz requested a review from swanandx October 30, 2026 02:49
@swanandx

swanandx commented Nov 6, 2026

Copy link
Copy Markdown
Member

this PR needs more work, needs a rebase once #7863 merges and keep the timeout inside its lock.

The timeout kills only uv. SIGKILL is not forwarded, so the Python child and its JVM keep running and overlap the retry. I reproduced this on Linux and macOS. Please start the run with start_new_session=True and os.killpg the group on any exception, not only on timeout. Keep the check=True behavior. The test stubs subprocess.run, so it cannot catch this. A real hanging script that starts a child would.

cc: @gz

@bestouff bestouff left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pushed a3c86b5 on top: the timed-out run is now started in its own session and the whole process group is killed, so the Spark JVM cannot outlive uv and overlap the retry. The unit tests stub _run_spark_subprocess instead of the subprocess module, and a new test runs a real child-with-grandchild tree and checks the grandchild dies with the timeout.

Still to do before merge: rebase once #7863 lands, keeping the timeout inside the global delta-spark lock.

@swanandx

Copy link
Copy Markdown
Member

#7863 is merged, this needs rebase / required work. @bestouff u plan to do it or shall i take it up?

@bestouff

Copy link
Copy Markdown
Contributor

Let me handle this

github-actions Bot and others added 2 commits November 13, 2026 09:22
`run_delta_spark` ran its `uv run ... python <script>` subprocess with no
timeout, and both it and `_build_delta_spark_fixture` retried only on
`CalledProcessError`.  A Spark process that hangs during Ivy dependency
resolution, rather than exiting non-zero, therefore held the test until the
pytest timeout killed it, with no retry.

Run the subprocess with a timeout (default 300s, about twice a healthy
cold build, so three attempts still fit well inside the pytest timeout)
and treat `TimeoutExpired` as retryable in both loops.

Signed-off-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com>
`subprocess.run(timeout=...)` SIGKILLs only its direct child, `uv`, which
cannot forward the signal. The `python` process and the JVM under it kept
running after a timeout, overlapping the retry: writing to the same table
in `test_delta_output_merge`, and into a staging dir the `finally` had
already removed in `_build_delta_spark_fixture`.

Run the command in its own session via `Popen(start_new_session=True)`
and `os.killpg` the group on every exit path, so nothing outlives the
attempt. `check=True` semantics are preserved.

The unit tests now stub `_run_spark_subprocess` instead of the whole
`subprocess` module. The old stubs had no `TimeoutExpired`, so any
exception in the guarded block surfaced as an AttributeError on the
stub rather than the real error.

Add a test that runs a real child-with-grandchild tree through the
runner and asserts the grandchild is gone after the timeout. It fails
when only the group leader is killed.
@bestouff
bestouff force-pushed the fred/delta-spark-subprocess-timeout branch from a3c86b5 to 41bb0c8 Compare November 13, 2026 08:24
@bestouff
bestouff added this pull request to the merge queue Nov 13, 2026
Merged via the queue into main with commit 3d952e2 Nov 13, 2026
1 check passed
@bestouff
bestouff deleted the fred/delta-spark-subprocess-timeout branch November 13, 2026 09:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants