← 输入框|github.com|刷新
Skip to content

[python] Size the Delta merge verifier's Spark job for a six-row table - #7947

Merged
ryzhyk merged 1 commit into
mainfrom
/merge-verify-spark-tasks
Nov 12, 2026
Merged

ryzhyk merged 1 commit into
mainfrom
/merge-verify-spark-tasks

Conversation

@ryzhyk

@ryzhyk ryzhyk commented Nov 12, 2026

Copy link
Copy Markdown
Contributor

On one thread, Spark's cluster-sized defaults turn merge_verify.py's three commands into more than 10,000 serial tasks, nearly all of them empty, so test_delta_spark_agrees_through_maintenance spends one to three minutes in its Delta Spark check, more under load.

The verifier sets the three settings that produce those tasks to 1: the parallelism VACUUM splits its file listing by (default 10,000), the shuffle partitions (200), and the partitions Delta rebuilds table state with (50). They change how the work is split, not what it reads.

Linked Issue

Describe Manual Test Plan

Checklist

  • Unit tests added/updated
  • Integration tests added/updated
  • Documentation updated
  • Changelog updated

Breaking Changes?

Mark if you think the answer is yes for any of these components:

Describe Incompatible Changes

On one thread, Spark's cluster-sized defaults turn merge_verify.py's three
commands into more than 10,000 serial tasks, nearly all of them empty, so
test_delta_spark_agrees_through_maintenance spends one to three minutes in its
Delta Spark check, more under load.

The verifier sets the three settings that produce those tasks to 1: the
parallelism VACUUM splits its file listing by (default 10,000), the shuffle
partitions (200), and the partitions Delta rebuilds table state with (50).
They change how the work is split, not what it reads.

Signed-off-by: Leonid Ryzhyk <ryzhyk@gmail.com>
@ryzhyk
ryzhyk requested a review from swanandx November 12, 2026 06:36
@mythical-fred-oss

Copy link
Copy Markdown
Contributor

LGTM — test-infra only (Spark config in merge_verify.py); no gate applies.

What I ran (built a 6-row Delta table with deletion vectors via delta-spark 4.x, local[1], then ran the verifier on identical copies):

wall time rows (initial / after_optimize / after_vacuum) VACUUM
before (settings stripped) 121 s 5 / 5 / 5 deleted 9 files
this PR 24 s 5 / 5 / 5, same set deleted 9 files
  • Same rows in every phase. Only the row order from collect() changed between runs, and test_delta_spark_agrees_through_maintenance sorts by id before it compares, so the order change cannot make the test flaky.
  • pre-commit run --files (ruff, ruff-format): passed.

@mihaibudiu mihaibudiu left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would have never found this solution

@ryzhyk

ryzhyk commented Nov 12, 2026

Copy link
Copy Markdown
Contributor Author

I would have never found this solution

Me neither

@ryzhyk
ryzhyk added this pull request to the merge queue Nov 12, 2026
Merged via the queue into main with commit 2bd519b Nov 12, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants