← 输入框|github.com|刷新
Skip to content

[connectors] Delta merge: get the key ranges of all files at once - #7948

Merged
swanandx merged 2 commits into
mainfrom
merge-bounds-measure
Nov 13, 2026
Merged

swanandx merged 2 commits into
mainfrom
merge-bounds-measure

Conversation

@swanandx

Copy link
Copy Markdown
Member

Linked Issue

None. To skip files, a merge flush uses the key range of each file. The key range is the smallest and the largest key in the file. Now the flush gets the ranges of all files with one call for each key column, not one call for each file.

before: for each file       -> get its key range
after:  for each key column -> get the key ranges of all files   (ROW keys: no change)
Table with 200k files Before After
Time to get the key ranges 1.6 s 25 ms
Time for one flush 2.6 s 0.9 s
Files skipped 199,936 199,936
Memory for each file, BIGINT key (50k files) 431 B 215 B

Describe Manual Test Plan

A new test makes sure that the new key ranges are equal to the old key ranges after DELETE, OPTIMIZE, VACUUM and a reopen. I ran the old code and the new code in turns to get the numbers in the table.

Checklist

  • Unit tests added/updated
  • Integration tests added/updated
  • Documentation updated
  • Changelog updated

Breaking Changes?

None.

@mythical-fred-oss

This comment was marked as resolved.

@swanandx
swanandx force-pushed the merge-bounds-measure branch from a69aaec to 81e5225 Compare November 12, 2026 12:03
@mythical-fred-oss

This comment was marked as resolved.

To skip files, a merge flush uses the key range of each file. The key
range is the smallest and the largest key in the file. Before this
change, the flush got the ranges one file at a time. Now it gets the
ranges of all files with one call for each key column. ROW keys still
use one call for each file, because delta-rs returns only the first
field of a ROW.

Signed-off-by: Swanand Mulay <73115739+swanandx@users.noreply.github.com>
@swanandx
swanandx force-pushed the merge-bounds-measure branch from 81e5225 to 71f1e92 Compare November 12, 2026 13:09
@mythical-fred-oss

Copy link
Copy Markdown
Contributor

LGTM.

What I ran: cargo test --locked -p dbsp_connectors --lib delta_table::merge passed 133 of 133, including the new log_bounds_match_the_per_file_view, row_bounds_stay_aligned_across_chunks and a_partition_key_column_takes_no_log_bound. cargo clippy --tests and rustfmt --check report nothing on the changed files.

What I checked against delta-rs 45a7cd6:

  • LogDataHandler::min_values/max_values and iter() both read self.data in the same order, so a file's row index points to its own bounds. The len == num_files guard protects against a mismatch.
  • round_up_ms gives the same result as ceil_datetime for µs timestamps, including negative values. Mins are not rounded on either path.
  • Binary and decimal columns, and files without stats, give no bound on both paths. Struct bounds still go through LogicalFileView, which applies the nested rounding.

@swanandx
swanandx requested a review from ryzhyk November 12, 2026 22:05
@ryzhyk

ryzhyk commented Nov 12, 2026

Copy link
Copy Markdown
Contributor

Not live bugs, but just FYI:

One gap worth closing (not a live bug). The kernel's extract_column_ref returns a column's min/max array without applying the null markers of its parent structs (stats_parsed, minValues). The old path's Scalar::from_array treated a null minValues as "no bound". Today's delta-rs and kernel also mark the value itself null in that case, which is why the tests matched. If a future delta-rs or kernel upgrade stopped doing that, a file without stats would get a garbage bound. That bound could skip a file holding the key and leave two live rows. The connector can't reach those parent structs through LogDataHandler, so the guard has to be a test. I suggest adding these stats shapes to log_bounds_match_the_per_file_view: {"numRecords":5,"minValues":null,"maxValues":null}, {"numRecords":5}, and min-only. Then such an upgrade fails CI instead of corrupting data.

and

The next bottleneck is outside this PR. The post-commit snapshot that merge::commit_actions adopts costs 5.4 µs per file at 32 columns, on every flush. That is about 50× the new bounds read. It grows with the column count, so delta-rs probably re-parses every file's stats on each commit (inferred, not profiled). The PR's remaining 0.9 s per flush fits this term at about 24 columns.

A file with stats but a null or missing minValues/maxValues must get
no key range, or merge can skip a file that holds the key. Cover the
JSON log and the stats_parsed checkpoint read, so a delta-rs or kernel
upgrade that drops the parent null fails here.

Signed-off-by: Swanand Mulay <73115739+swanandx@users.noreply.github.com>
let column = Column::from_name(name);
let files = log.num_files();
(
usable(log.min_values(&column), files),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pitfall (medium): pick_stats turns every failure into None (evaluator build error, a missing field, a concat error), and usable drops length mismatches without a word. Before this change a bad stat lost pruning for one file. Now it loses pruning for every file on that column, and the flush goes from about 1 s to a full scan with no log line. Could you add a warn! (once per flush) when a non-partition, non-binary key column comes back None or with the wrong length? Otherwise a delta-rs bump that breaks this path only shows up as a slow pipeline.

@mythical-fred-oss

Copy link
Copy Markdown
Contributor

What I ran

Check Result
cargo test --locked -p dbsp_connectors --lib delta_table::merge ✅ 133 passed
cargo clippy -p dbsp_connectors --tests (changed files) ✅ clean
rustfmt --check on changed files ✅
delta-rs pick_stats / ceil_datetime at the pinned rev ✅ same order as log_data().iter(); round-up matches for negative values; nanosecond-timestamps is off, so µs-only round_up_ms is enough

The equivalence test (old view vs. new bounds across checkpoint, DELETE, OPTIMIZE, VACUUM, reopen, partial or absent stats, and ROW chunk edges) is thorough, and it pins down both the null-parent case and the alignment case.

Missing (medium): SQL type coverage. This is a connector change. log_bounds_match_the_per_file_view has no FLOAT/DOUBLE, no MAP, and no TIME or unsigned columns (the types Feldera writes for those to Delta). The per-column log.min_values path is new for each of them, so please add these columns to the schema in typed_batch. The expected usable set then shows whether each one prunes.

Uncovered (low): the usable() path where the length differs from num_files, and mixed-case column names, are not tested.

No other gate fails: no unsafe code, no new dependencies, no docs needed (internal only), and the commit messages are good.

@swanandx
swanandx added this pull request to the merge queue Nov 13, 2026
Merged via the queue into main with commit 1a4a1ab Nov 13, 2026
1 check passed
@swanandx
swanandx deleted the merge-bounds-measure branch November 13, 2026 11:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants