Repository navigation
[connectors] Delta merge: get the key ranges of all files at once - #7948
Conversation
This comment was marked as resolved.
This comment was marked as resolved.
a69aaec to
81e5225
Compare
This comment was marked as resolved.
This comment was marked as resolved.
To skip files, a merge flush uses the key range of each file. The key range is the smallest and the largest key in the file. Before this change, the flush got the ranges one file at a time. Now it gets the ranges of all files with one call for each key column. ROW keys still use one call for each file, because delta-rs returns only the first field of a ROW. Signed-off-by: Swanand Mulay <73115739+swanandx@users.noreply.github.com>
81e5225 to
71f1e92
Compare
|
LGTM. What I ran: What I checked against delta-rs
|
|
Not live bugs, but just FYI: and |
A file with stats but a null or missing minValues/maxValues must get no key range, or merge can skip a file that holds the key. Cover the JSON log and the stats_parsed checkpoint read, so a delta-rs or kernel upgrade that drops the parent null fails here. Signed-off-by: Swanand Mulay <73115739+swanandx@users.noreply.github.com>
| let column = Column::from_name(name); | ||
| let files = log.num_files(); | ||
| ( | ||
| usable(log.min_values(&column), files), |
There was a problem hiding this comment.
pitfall (medium): pick_stats turns every failure into None (evaluator build error, a missing field, a concat error), and usable drops length mismatches without a word. Before this change a bad stat lost pruning for one file. Now it loses pruning for every file on that column, and the flush goes from about 1 s to a full scan with no log line. Could you add a warn! (once per flush) when a non-partition, non-binary key column comes back None or with the wrong length? Otherwise a delta-rs bump that breaks this path only shows up as a slow pipeline.
|
What I ran
The equivalence test (old view vs. new bounds across checkpoint, DELETE, OPTIMIZE, VACUUM, reopen, partial or absent stats, and ROW chunk edges) is thorough, and it pins down both the null-parent case and the alignment case. Missing (medium): SQL type coverage. This is a connector change. Uncovered (low): the No other gate fails: no unsafe code, no new dependencies, no docs needed (internal only), and the commit messages are good. |
Linked Issue
None. To skip files, a merge flush uses the key range of each file. The key range is the smallest and the largest key in the file. Now the flush gets the ranges of all files with one call for each key column, not one call for each file.
Describe Manual Test Plan
A new test makes sure that the new key ranges are equal to the old key ranges after DELETE, OPTIMIZE, VACUUM and a reopen. I ran the old code and the new code in turns to get the numbers in the table.
Checklist
Breaking Changes?
None.