Repository navigation
[connectors] Bump arrow-rs to 59.3.2. - #7895
Conversation
parquet 58.3 to 59.3.1 cannot skip rows inside a DELTA_BINARY_PACKED miniblock of more than 64 values. Databricks Photon writes 256-value miniblocks for tables with delta.parquet.format.version = 2.12.0, so applying a deletion vector to one of their files failed with "cannot skip miniblock of size 256", and the Delta input connector retried the file forever. 59.3.2 has the upstream fix (apache/arrow-rs#11021). Every arrow-rs crate moves to it, since they are released together, and Cargo.toml raises the parquet floor to it. A new test reads a Photon-written file through the deletion-vector reader, so it fails if the lockfile picks up a parquet without the fix. Fixes #7091 Signed-off-by: Leonid Ryzhyk <ryzhyk@gmail.com>
The Photon fixture packs every miniblock at bit width 0, so its test reaches only the decoder's zero-width skip path. The new test reads a file whose 256-value miniblocks are packed at 9 to 11 bits through the deletion-vector reader, with skips inside a miniblock, across miniblocks, and across blocks. The parquet crate's writer puts at most 64 values in a miniblock, so the test encodes the page itself and lets the crate's column writer place it in a file. Like the Photon test, it fails on parquet 58.3 to 59.3.1; it also fails on 58.2, which panics on a long skip inside a bit-packed miniblock. Signed-off-by: Leonid Ryzhyk <ryzhyk@gmail.com>
|
Review: LGTM. This is a patch bump (arrow-rs 59.3.0 → 59.3.2, all crates from one release, Apache-2.0) with tests that fail without the fix.
|
swanandx
left a comment
There was a problem hiding this comment.
PR LGTM! as per my friend, there can be small improvements:
It ran the new tests. They pass on parquet 59.3.2. Both fail on 59.3.0 with cannot skip miniblock of size 256. It compared the arrow-rs source of 59.3.0 and 59.3.2. The changes are bug fixes only.
-
WideMiniblockPages::write_pagedoes not change a page that is not a V1DataPage. With V2 pages, the file has the usual 64-value miniblocks. Then the test passes on the bad parquet. It addedset_writer_version(PARQUET_2_0), and the test passed on 59.3.0. Make the test fail on a page that is not a V1 data page or a dictionary page. Or, setPARQUET_1_0and make sure that the test changed one page. -
Small: parquet skips rows only when a row selection has 32 or more rows per run on average. With shorter runs, it reads all rows. The current bitmaps have longer runs. Add one comment line, so that a change to a bitmap does not remove the coverage.
-
Small: the Photon test file has our Databricks cluster id and job group id. They are not secrets. You can remove them.
With V2 data pages, the page rewriter would pass the writer's 64-value miniblocks through, and the bit-packed test would pass on parquet 59.3.0. With bitmap runs shorter than 32 rows on average, parquet decodes the rows between selected ones and filters them out instead of skipping them. The bit-packed test now writes V1 data pages explicitly, its page rewriter panics on any other page, and the shared read helper documents the 32-row minimum. Signed-off-by: Leonid Ryzhyk <ryzhyk@gmail.com>
|
I addressed 1 and 2. I don't think 3 is an issue. |
parquet 58.3 to 59.3.1 cannot skip rows inside a DELTA_BINARY_PACKED
miniblock of more than 64 values. Databricks Photon writes 256-value
miniblocks for tables with delta.parquet.format.version = 2.12.0, so
applying a deletion vector to one of their files failed with "cannot
skip miniblock of size 256", and the Delta input connector retried the
file forever.
59.3.2 has the upstream fix (apache/arrow-rs#11021). Every arrow-rs
crate moves to it, since they are released together, and Cargo.toml
raises the parquet floor to it. A new test reads a Photon-written file
through the deletion-vector reader, so it fails if the lockfile picks
up a parquet without the fix.
Linked Issue
Closes #7091
Describe Manual Test Plan
Checklist
Breaking Changes?
Mark if you think the answer is yes for any of these components:
Describe Incompatible Changes