Repository navigation
[adapters] Delta: let a filter name a nested column-mapped field - #7669
Conversation
|
@swanandx This is outside my expertise so I wasn't able to effectively review it myself. I think you know more about this. @anandbraman Does this make sense relative to the bug report? |
swanandx
left a comment
There was a problem hiding this comment.
LGTM, thanks!
we can now remove the nested_map variable and the relabel_nested_columns(&batch, &nested_map) call in execute_df_inner as it shouldn't be required with this fix in place. RelabelNestedUdf already renames nested fields before the query runs, so that step no longer changes anything
also, we should rebase and Sign off the commit before merging
thank you so much for the fix!
| # - "round-robin pre-split unpartitioned scans": reads a many-file table in | ||
| # parallel (#2042). `DELTA_SCAN_ROUND_ROBIN=false` turns it off. | ||
| # Upstream now, do not reapply: the delta-io/delta-rs#4692 backport. | ||
| deltalake = { git = "https://github.com/feldera/delta-rs.git", rev = "9d996253cc60e5a6f35b2be5aef1cce89e08d951" } |
There was a problem hiding this comment.
why we changing this? rebase should solve this
There was a problem hiding this comment.
Right, that was an artifact of an old base. The branch is now rebased onto main as one signed-off commit, and it no longer touches Cargo.toml or Cargo.lock. That commit also drops the per-batch relabel_nested_columns call and nested_map from execute_df_inner, as you suggested. The delta_table tests pass.
(Claude, on behalf of @blp)
0939660 to
b17f5af
Compare
When the connector reads Parquet files itself (distributed snapshot, follow, change data feed, CDC), it reads them with physical `col-<uuid>` names at every level. `project_physical_to_logical` put back only the top-level logical names, so a `filter`, `snapshot_filter`, `cdc_delete_filter`, or `cdc_order_by` that named a nested field failed with "Field ... not found in struct". The nested names were put back only per batch, after the query had run. `project_physical_to_logical` now also puts back nested names, through a scalar function that applies `relabel_array` to each column that holds column-mapped nested fields. This makes the per-batch relabel in `execute_df_inner` redundant, so it is gone. Fixes #7657. Signed-off-by: Ben Pfaff <blp@feldera.com>
b17f5af to
e89cf5d
Compare
|
Verdict: PR bug, high confidence. The new test Evidence
Fix: renamed the test to Take it with: Compare: delta-nested-filter...fred/observe-7669-41500486043 |
…me limit. Signed-off-by: Ben Pfaff <blp@feldera.com>
Linked Issue
Fixes #7657.
When the Delta input connector reads Parquet files itself, it reads them with physical
col-<uuid>names at every level.project_physical_to_logicalput back only the top-level logical names, so afilter,snapshot_filter,cdc_delete_filter, orcdc_order_bythat named a nested field failed withField ... not found in struct. The nested names were put back only per batch, after the query had run.distributed: false(delta-rs provider)distributed: trueproject_physical_to_logicalnow also puts back nested names, through a scalar function (RelabelNestedUdf) that applies the existingrelabel_arrayto each column that holds column-mapped nested fields. It is zero-copy and keeps struct nulls.A predicate on a nested field cannot push down through the function, so it gets no row-group pruning. Before this change those predicates failed, so no pruning is lost.
Describe Manual Test Plan
cargo test -p dbsp_adapters --lib delta_table: 430 passed.cargo clippy -p dbsp_adapters --all-targets: clean.test_delta_input_column_mapping_nested_dist_filterfor the distributed snapshot case. I have not run it locally; CI will.Checklist
Breaking Changes?
None.