Repository navigation
[connectors] Update iceberg-rust to 4dd8c946 - #7721
Conversation
|
What I ran
Pitfall (medium): unrelated lockfile downgrades. Besides the iceberg and avro bumps, Observation: iceberg brings in new transitive deps ( Not covered: no Feldera-side test checks that a scan does not stall when many data files are open; the only coverage is the upstream test. Otherwise LGTM. |
2236dc2 to
ccb82b9
Compare
|
What I ran:
|
|
fred is right, fixing this |
|
should be fixed and safe to bump after #7735 |
ccb82b9 to
7365f03
Compare
|
What I ran
I could not run missing (high): no changelog entry for a declared breaking change. The PR body has a good table and says follow pipelines must be restarted from a new snapshot, but none of this is in missing (medium): no test for the changed behaviour.
Cases 1 and 2 fit next to the all-types test in |
7365f03 to
7acb094
Compare
|
Review of the iceberg-rust bump to 4dd8c946 What I ran:
Once the changelog entry with migration guidance and the tests for cases (1) and (2) are in, I'm happy to approve. |
2ec81bd to
fb2aac7
Compare
|
What I ran
New crates in the lock file ( Missing (medium): no tests for the new scan behaviour. The bump changes how DataFusion pushes down filters: LIKE prefixes are now unescaped, and Parquet pages are pruned with the page index. Both change which rows a +/// A `LIKE` prefix filter is pushed down to the scan and must keep exactly the matching rows.
+#[engine_test(cfg(feature = "iceberg-tests-fs"))]
+pub fn localfs_input_test_like_filter(engine: &dyn PipelineTestEngine) {
+ iceberg_localfs_input_test(
+ engine,
+ 100_000,
+ json!({ "snapshot_filter": "s LIKE 's1%' AND NOT (i IN (1, 10, 100))" }),
+ &|x| {
+ let i = x["i"].as_i64().unwrap();
+ x["s"].as_str().unwrap().starts_with("s1") && ![1, 10, 100].contains(&i)
+ },
+ );
+}The test data has no literal Missing (medium): the other side of the breaking change has no test. The changelog says that a table with no name mapping now reads nested columns as NULL. Nothing tests that path. A Pitfall (low): the commit says apache-avro 0.22 needs |
fb2aac7 to
b1c10c7
Compare
|
Skipping the other points:
|
| field ids, such as files added with `add_files`, now read their real values | ||
| instead of NULL. If the table has no name mapping, these columns now read as | ||
| NULL. A follow pipeline that read such a file before the upgrade can show | ||
| wrong rows when the file is deleted later. |
There was a problem hiding this comment.
pitfall (medium): This entry tells users that a follow pipeline can show wrong rows, but not what to do about it. Add the action, e.g. "if a follow pipeline read add_files data before upgrading, restart it from a fresh snapshot (clear its storage/checkpoint)". Also, "these columns now read as NULL" leaves out what they read before (nothing? an error? positional values?). Say what changed so users can tell whether their data is affected.
|
What I ran
missing (medium): tests for the new read-path behaviour. The commit says reads now unescape LIKE prefixes before pushdown and prune pages with the page index. No test in the repo covers either one; the PR only mentions a local check of 22 filters. I added a LIKE + NOT snapshot filter locally and it passes: +#[engine_test(cfg(feature = "iceberg-tests-fs"))]
+pub fn localfs_input_test_like_filter(engine: &dyn PipelineTestEngine) {
+ iceberg_localfs_input_test(
+ engine,
+ 100_000,
+ json!({ "snapshot_filter": "s LIKE 's12%' AND NOT (i < 1250)" }),
+ &|x| x["s"].as_str().unwrap().starts_with("s12") && x["i"].as_i64().unwrap() >= 1250,
+ );
+}That test doesn't catch an unescape bug, though, because the test strings contain no Cases not covered by the PR's tests
|
b1c10c7 to
717ad65
Compare
| field ids, such as files added with `add_files`, now read their real values | ||
| instead of NULL. If the table has no name mapping, these columns now read as | ||
| NULL instead of values matched by position. A follow pipeline that read such | ||
| a file before the upgrade can show wrong rows when the file is deleted later. |
There was a problem hiding this comment.
pitfall (medium): This tells users a follow pipeline can show wrong rows, but not what to do about it. Add the action, e.g. "restart affected follow pipelines from a fresh snapshot (clear storage) after upgrading". Also, nested columns that silently turn NULL when there's no name mapping are hard to notice. Could the connector log a warning once per file when it falls back to NULL for want of field ids and a name mapping?
|
What I ran
Missing tests (medium)
I didn't check the upstream iceberg-rust diff line by line. I relied on the fork now matching upstream and on the test results above. |
| field ids, such as files added with `add_files`, now read their real values | ||
| instead of NULL. If the table has no name mapping, these columns now read as | ||
| NULL instead of values matched by position. A follow pipeline that read such | ||
| a file before the upgrade can show wrong rows when the file is deleted later. |
There was a problem hiding this comment.
missing (medium): this names two silent data changes but gives the user nothing to do about them.
- Follow pipelines: say what to do, e.g. "rebuild the pipeline from a fresh snapshot (clear storage) after upgrading if the table has files without field ids".
- Tables without a name mapping: say how to fix it, e.g. "set
schema.name-mapping.defaulton the table". Better still, have the connector log a warning when it opens such a table, so the NULLs are not silent.
|
What I ran
Gates: tests, docs (changelog) and licences (new transitive crates are MIT/Apache) pass. The apache-avro 0.22 bump comes in only through iceberg; Feldera's Avro format stays on 0.18. Missing (medium): test cases not covered
Tech-debt (low): |
d3f1a14 to
8ef5eae
Compare
8ef5eae to
14f9884
Compare
14f9884 to
75d0c64
Compare
Verdict: flaky test, not this PR — merge queue should pass on re-queueFailed job: Adapter Tests / Test Adapters (amd64)
Already tracked as a known flake: #7820 (filed yesterday from a different PR, same assertion, same line, arm64 runner). This run reproduces it on amd64 with the same preceding burst of Next step: no fix needed here; re-queueing the PR should pass. If it fails again on this same test, that's issue #7820, not a new problem. |
|
Verdict: flaky test, not this PR. Merge queue run failed in
This PR's diff only touches Next step: re-queueing should pass — nothing in the failure touches code this PR changed. |
|
If the fork now matches upstream, why are we still using the fork? |
The fork now matches upstream, which merged our reader stall fix. The DataFusion integration crate is renamed back to iceberg-datafusion. Its snapshot reads now run as a DataSource, prune Parquet pages with the page index, and unescape LIKE prefixes before pushdown. Nested columns in files without field ids now read their real values. A new test reads such a table, with a page index, under 9 filters. The changelog notes the change. apache-avro 0.22 needs serde_json 1.0.150 or later. The JSON text of floats does not change because of #7735. serde_json now writes 1e40 as 1e+40 in parsed number text, so one test error text changes. Signed-off-by: Swanand Mulay <73115739+swanandx@users.noreply.github.com>
75d0c64 to
9291b9f
Compare
upstream release process is quite slow, and lot of times we want the latest main ( say to use bumped dependecies ) or have some patches until our PRs get accepted upstream. hence sticking to fork |
Describe Manual Test Plan
The fork now matches upstream with no patches. The Iceberg unit, file system
and follow tests pass, and so do the Delta and Parquet tests.
The new test
localfs_input_test_add_filesreads a table made withadd_files,with a page index, under 9 filters. It fails when the upstream nested-column fix
is reverted, and when page pruning or the LIKE unescape fix is broken.
Breaking Changes?
Nested columns in data files without field ids, such as files added with
add_files, now read their real values instead of NULL. If the table has noname mapping, these columns now read as NULL instead of values matched by
position. A follow pipeline that read such a file before the upgrade can show
wrong rows when the file is deleted later. The changelog says the same.