Repository navigation
[python] Size the Delta merge verifier's Spark job for a six-row table - #7947
Merged
Merged
Conversation
On one thread, Spark's cluster-sized defaults turn merge_verify.py's three commands into more than 10,000 serial tasks, nearly all of them empty, so test_delta_spark_agrees_through_maintenance spends one to three minutes in its Delta Spark check, more under load. The verifier sets the three settings that produce those tasks to 1: the parallelism VACUUM splits its file listing by (default 10,000), the shuffle partitions (200), and the partitions Delta rebuilds table state with (50). They change how the work is split, not what it reads. Signed-off-by: Leonid Ryzhyk <ryzhyk@gmail.com>
Contributor
|
LGTM — test-infra only (Spark config in What I ran (built a 6-row Delta table with deletion vectors via delta-spark 4.x,
|
mihaibudiu
reviewed
Nov 12, 2026
mihaibudiu
left a comment
Contributor
There was a problem hiding this comment.
I would have never found this solution
swanandx
approved these changes
Nov 12, 2026
Contributor
Author
Me neither |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
On one thread, Spark's cluster-sized defaults turn merge_verify.py's three commands into more than 10,000 serial tasks, nearly all of them empty, so test_delta_spark_agrees_through_maintenance spends one to three minutes in its Delta Spark check, more under load.
The verifier sets the three settings that produce those tasks to 1: the parallelism VACUUM splits its file listing by (default 10,000), the shuffle partitions (200), and the partitions Delta rebuilds table state with (50). They change how the work is split, not what it reads.
Linked Issue
Describe Manual Test Plan
Checklist
Breaking Changes?
Mark if you think the answer is yes for any of these components:
Describe Incompatible Changes