← 输入框|github.com|刷新
Skip to content

[connectors] Bump arrow-rs to 59.3.2. - #7895

Merged
ryzhyk merged 3 commits into
mainfrom
issue-7091
Nov 9, 2026
Merged

ryzhyk merged 3 commits into
mainfrom
issue-7091

Conversation

@ryzhyk

@ryzhyk ryzhyk commented Nov 9, 2026

Copy link
Copy Markdown
Contributor

parquet 58.3 to 59.3.1 cannot skip rows inside a DELTA_BINARY_PACKED
miniblock of more than 64 values. Databricks Photon writes 256-value
miniblocks for tables with delta.parquet.format.version = 2.12.0, so
applying a deletion vector to one of their files failed with "cannot
skip miniblock of size 256", and the Delta input connector retried the
file forever.

59.3.2 has the upstream fix (apache/arrow-rs#11021). Every arrow-rs
crate moves to it, since they are released together, and Cargo.toml
raises the parquet floor to it. A new test reads a Photon-written file
through the deletion-vector reader, so it fails if the lockfile picks
up a parquet without the fix.

Linked Issue

Closes #7091

Describe Manual Test Plan

Checklist

  • Unit tests added/updated
  • Integration tests added/updated
  • Documentation updated
  • Changelog updated

Breaking Changes?

Mark if you think the answer is yes for any of these components:

Describe Incompatible Changes

parquet 58.3 to 59.3.1 cannot skip rows inside a DELTA_BINARY_PACKED
miniblock of more than 64 values. Databricks Photon writes 256-value
miniblocks for tables with delta.parquet.format.version = 2.12.0, so
applying a deletion vector to one of their files failed with "cannot
skip miniblock of size 256", and the Delta input connector retried the
file forever.

59.3.2 has the upstream fix (apache/arrow-rs#11021). Every arrow-rs
crate moves to it, since they are released together, and Cargo.toml
raises the parquet floor to it. A new test reads a Photon-written file
through the deletion-vector reader, so it fails if the lockfile picks
up a parquet without the fix.

Fixes #7091

Signed-off-by: Leonid Ryzhyk <ryzhyk@gmail.com>
The Photon fixture packs every miniblock at bit width 0, so its test
reaches only the decoder's zero-width skip path. The new test reads a
file whose 256-value miniblocks are packed at 9 to 11 bits through the
deletion-vector reader, with skips inside a miniblock, across
miniblocks, and across blocks.

The parquet crate's writer puts at most 64 values in a miniblock, so the
test encodes the page itself and lets the crate's column writer place it
in a file. Like the Photon test, it fails on parquet 58.3 to 59.3.1; it
also fails on 58.2, which panics on a long skip inside a bit-packed
miniblock.

Signed-off-by: Leonid Ryzhyk <ryzhyk@gmail.com>
@ryzhyk
ryzhyk requested a review from swanandx November 9, 2026 01:27
@ryzhyk ryzhyk added the connectors Issues related to the adapters/connectors crate label Nov 9, 2026
@mythical-fred-oss

Copy link
Copy Markdown
Contributor

Review: LGTM. This is a patch bump (arrow-rs 59.3.0 → 59.3.2, all crates from one release, Apache-2.0) with tests that fail without the fix.

What I ran Result
cargo test --locked -p dbsp_connectors --lib masked_reader ✅ 4/4, including both new tests
Same tests after pinning parquet back to 59.3.0 ❌ both fail with cannot skip miniblock of size 256, so the tests catch the regression
cargo test --locked -p dbsp_connectors --lib deletion_vector ✅ 39/39
pre-commit run --files <changed files> ✅ all hooks pass
  • missing (medium): the new tests cover only a REQUIRED INT64 column. The bug also hits other cases: Photon delta-encodes INT32 columns too, and the old 58.2 panic sized its buffer by physical type (32 for INT32). A nullable column, where definition levels shift row and value positions, is also untested. write_wide_miniblock_parquet could take the physical type and nullability as parameters, which would cover both with little extra code.
  • Gates: no unsafe code, no new dependencies, no breaking changes. The changelog has an entry, and no other docs are needed. The tests are deterministic. The Unity stress test is unblocked, but it still needs credentials to run, so CI does not exercise it.

@swanandx swanandx left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PR LGTM! as per my friend, there can be small improvements:

It ran the new tests. They pass on parquet 59.3.2. Both fail on 59.3.0 with cannot skip miniblock of size 256. It compared the arrow-rs source of 59.3.0 and 59.3.2. The changes are bug fixes only.

  1. WideMiniblockPages::write_page does not change a page that is not a V1 DataPage. With V2 pages, the file has the usual 64-value miniblocks. Then the test passes on the bad parquet. It added set_writer_version(PARQUET_2_0), and the test passed on 59.3.0. Make the test fail on a page that is not a V1 data page or a dictionary page. Or, set PARQUET_1_0 and make sure that the test changed one page.

  2. Small: parquet skips rows only when a row selection has 32 or more rows per run on average. With shorter runs, it reads all rows. The current bitmaps have longer runs. Add one comment line, so that a change to a bitmap does not remove the coverage.

  3. Small: the Photon test file has our Databricks cluster id and job group id. They are not secrets. You can remove them.

With V2 data pages, the page rewriter would pass the writer's 64-value
miniblocks through, and the bit-packed test would pass on parquet
59.3.0. With bitmap runs shorter than 32 rows on average, parquet
decodes the rows between selected ones and filters them out instead of
skipping them.

The bit-packed test now writes V1 data pages explicitly, its page
rewriter panics on any other page, and the shared read helper documents
the 32-row minimum.

Signed-off-by: Leonid Ryzhyk <ryzhyk@gmail.com>
@ryzhyk
ryzhyk added this pull request to the merge queue Nov 9, 2026
@ryzhyk

ryzhyk commented Nov 9, 2026

Copy link
Copy Markdown
Contributor Author

I addressed 1 and 2. I don't think 3 is an issue.

Merged via the queue into main with commit 5857c6c Nov 9, 2026
1 check passed
@ryzhyk
ryzhyk deleted the issue-7091 branch November 9, 2026 22:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

connectors Issues related to the adapters/connectors crate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[connectors] Delta connector is affected by a parquet bug.

2 participants