Repository navigation
[python] Time out and retry hung Delta Spark subprocesses - #7726
Merged
Merged
Conversation
Member
|
this PR needs more work, needs a rebase once #7863 merges and keep the timeout inside its lock. cc: @gz |
bestouff
approved these changes
Nov 9, 2026
bestouff
left a comment
Contributor
There was a problem hiding this comment.
Pushed a3c86b5 on top: the timed-out run is now started in its own session and the whole process group is killed, so the Spark JVM cannot outlive uv and overlap the retry. The unit tests stub _run_spark_subprocess instead of the subprocess module, and a new test runs a real child-with-grandchild tree and checks the grandchild dies with the timeout.
Still to do before merge: rebase once #7863 lands, keeping the timeout inside the global delta-spark lock.
Member
Contributor
|
Let me handle this |
`run_delta_spark` ran its `uv run ... python <script>` subprocess with no timeout, and both it and `_build_delta_spark_fixture` retried only on `CalledProcessError`. A Spark process that hangs during Ivy dependency resolution, rather than exiting non-zero, therefore held the test until the pytest timeout killed it, with no retry. Run the subprocess with a timeout (default 300s, about twice a healthy cold build, so three attempts still fit well inside the pytest timeout) and treat `TimeoutExpired` as retryable in both loops. Signed-off-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com>
`subprocess.run(timeout=...)` SIGKILLs only its direct child, `uv`, which cannot forward the signal. The `python` process and the JVM under it kept running after a timeout, overlapping the retry: writing to the same table in `test_delta_output_merge`, and into a staging dir the `finally` had already removed in `_build_delta_spark_fixture`. Run the command in its own session via `Popen(start_new_session=True)` and `os.killpg` the group on every exit path, so nothing outlives the attempt. `check=True` semantics are preserved. The unit tests now stub `_run_spark_subprocess` instead of the whole `subprocess` module. The old stubs had no `TimeoutExpired`, so any exception in the guarded block surfaced as an AttributeError on the stub rather than the real error. Add a test that runs a real child-with-grandchild tree through the runner and asserts the grandchild is gone after the timeout. It fails when only the group leader is killed.
bestouff
force-pushed
the
fred/delta-spark-subprocess-timeout
branch
from
November 13, 2026 08:24
a3c86b5 to
41bb0c8
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
run_delta_spark(python/tests/utils.py) runs its Spark subprocess with no timeout. Both retry loops (there and in_build_delta_spark_fixture) catch onlyCalledProcessError. So when Spark hangs in Ivy dependency resolution instead of exiting, nothing retries it, and the test blocks until pytest-timeout kills it.Changes
run_delta_sparktimeout_s=300parameter, passed tosubprocess.run(timeout=...)TimeoutExpiredas well asCalledProcessError, and log the exceptiontests/unit/test_delta_fixture_build_lock.pytest_hung_builder_is_timed_out_and_retried300s is about twice a healthy cold build. Three attempts plus backoff come to about 915s, which still fits under the 1500s pytest timeout.
Testing
uv run pytest tests/unit/test_delta_fixture_build_lock.py: 5 passed.utils.pychange reverted, the new test fails ("the builder subprocess must be run with a timeout").pre-commit run --files <changed files>: ruff and ruff-format pass.