Repository navigation
[iceberg] Read deletion vectors in follow mode - #7590
Conversation
|
What I ran
missing (medium): unit coverage. The happy path of diff@@ -3723,6 +3723,53 @@ mod deletion_vector_tests {
})
.contains("two deletion vectors"));
}
+
+ #[test]
+ fn pairs_deletion_vectors_with_their_data_files() {
+ let paired = pair_deletion_vectors(
+ 7,
+ SnapshotFiles {
+ added: vec![data_file("new.parquet")],
+ removed: vec![data_file("old.parquet")],
+ added_dvs: vec![
+ dv("n.puffin", "new.parquet"),
+ dv("k2.puffin", "kept.parquet"),
+ ],
+ removed_dvs: vec![
+ dv("o.puffin", "old.parquet"),
+ dv("k1.puffin", "kept.parquet"),
+ ],
+ },
+ )
+ .unwrap();
+ assert_eq!(paired.added.len(), 1);
+ assert!(paired.added[0].deletion_vector.is_some());
+ assert_eq!(paired.removed.len(), 1);
+ assert_eq!(
+ paired.removed[0].deletion_vector.as_ref().unwrap().file_path,
+ "o.puffin"
+ );
+ assert_eq!(paired.new_deletes.len(), 1);
+ let nd = &paired.new_deletes[0];
+ assert_eq!(nd.data_file_path, "kept.parquet");
+ assert_eq!(nd.replaced.as_ref().unwrap().data_file.file_path(), "k1.puffin");
+ }
+
+ #[test]
+ fn select_positions_keeps_only_selected_rows() {
+ use datafusion::arrow::array::{Int64Array, StringArray, ArrayRef};
+ let batch = RecordBatch::try_from_iter([
+ ("v", Arc::new(StringArray::from(vec!["a", "b", "c", "d"])) as ArrayRef),
+ ("_file", Arc::new(StringArray::from(vec!["f"; 4])) as ArrayRef),
+ ("_pos", Arc::new(Int64Array::from(vec![10, 11, 12, 13])) as ArrayRef),
+ ])
+ .unwrap();
+ let selected = HashMap::from([("f".to_string(), Arc::new(RoaringTreemap::from_iter([11u64, 13])))]);
+ let out = select_positions(batch, &selected).unwrap();
+ assert_eq!(out.num_columns(), 1);
+ let v = out.column(0).as_string::<i32>();
+ assert_eq!(v.iter().flatten().collect::<Vec<_>>(), ["b", "d"]);
+ }
}
#[cfg(test)]Uncovered cases (medium):
Gate: SQL type coverage (medium). The new The other gates are fine: docs are updated, the new deps are MIT/Apache, there is no |
|
What I ran
Gates
Missing test cases (medium)
Logic, docs, licences ( |
d368867 to
864fe23
Compare
|
What I ran
Gates: tests ✅, docs ✅, breaking change described in the PR and docs ✅. New deps Test cases the PR does not cover (medium):
|
864fe23 to
39c2047
Compare
|
What I ran
The pairing logic, spec reasoning, docs and breaking-change note look right. Licenses are fine:
QA tests I added locally (pass)+ #[test]
+ fn qa_rejects_trailing_bytes_after_bitmap() {
+ let mut body = vec![0xD1, 0xD3, 0x39, 0x64];
+ RoaringTreemap::from_iter([1, 2]).serialize_into(&mut body).unwrap();
+ body.push(0);
+ let mut blob = (body.len() as u32).to_be_bytes().to_vec();
+ blob.extend_from_slice(&body);
+ blob.extend_from_slice(&crc32fast::hash(&body).to_be_bytes());
+ let e = decode_deletion_vector(&blob).unwrap_err().to_string();
+ assert!(e.contains("follow the bitmap"), "{e}");
+ }
+
+ #[test]
+ fn qa_pairs_plain_files_and_first_dv() {
+ let paired = pair_deletion_vectors(1, SnapshotFiles {
+ added: vec![data_file("a.parquet")],
+ removed: vec![data_file("r.parquet")],
+ added_dvs: vec![dv("x.puffin", "kept.parquet")],
+ removed_dvs: vec![],
+ }).unwrap();
+ assert!(paired.added[0].deletion_vector.is_none());
+ assert!(paired.removed[0].deletion_vector.is_none());
+ assert!(paired.new_deletes[0].replaced.is_none());
+ pair_deletion_vectors(2, SnapshotFiles {
+ removed: vec![data_file("f.parquet")],
+ removed_dvs: vec![dv("x.puffin", "f.parquet")],
+ ..Default::default()
+ }).unwrap();
+ }No hard gate fails. LGTM once the |
39c2047 to
9ed66a3
Compare
|
What I ran
Gates: tests, docs, breaking-change notice, licences ( Missing tests (medium):
Pitfall (medium): a writer that removes a data file but keeps its DV breaks the spec. The connector then silently emits deletes for rows it never inserted (negative weights downstream), and nothing fails. The docs call this out. Nothing cheap can detect it, so please describe this symptom (negative weights after a copy-on-write change) in the docs, so users can diagnose it. Otherwise LGTM once the retry question is resolved. |
9ed66a3 to
8eab915
Compare
|
Ran
Gates
Missing coverage (medium): - replaced: Some(write_dv("old.puffin", old, old.len() as u64)),
+ replaced: (!old.is_empty()).then(|| write_dv("old.puffin", old, old.len() as u64)),
...
+ #[tokio::test]
+ async fn new_deletes_read_without_old_vector_selects_all_positions() {
+ let selected = newly_deleted(&[], &[0, 2, 1 << 33], 3).await.unwrap();
+ assert_eq!(selected, Some(vec![0, 2, 1 << 33]));
+ assert_eq!(newly_deleted(&[], &[], 0).await.unwrap(), None);
+ }Other cases with no test:
|
8eab915 to
870511b
Compare
870511b to
61cbc61
Compare
|
LGTM. I traced the DV pairing (added / removed / kept file with a replaced DV) against the spec and found no correctness bugs. Docs are updated, the commit message is good, and the new deps ( What I ran
pitfall (medium): the e2e fixture depends on the network. Cases the PR's tests don't cover (none of them block):
|
61cbc61 to
466a6e0
Compare
|
Review: LGTM. I found no correctness bugs, and every feldera-pr-checks gate passes: tests, docs, commit message, deps ( What I ran
Cases I checked by reading the code: added file + DV, removed file + old DV, kept file with first / replacement DV, DV that shrinks (rejected), DV removed without replacement (rejected), CoW rewrite of a file with a DV, compaction skipped, V2 file left over from before the start snapshot (caught on the first delete). Type coverage is fine: the new path only filters batches, and Missing tests (medium):
+ #[test]
+ fn classifies_delete_files() {
+ let file = |content, format| {
+ DataFileBuilder::default().file_path("d".to_string()).file_format(format)
+ .content(content).record_count(1).partition(Struct::empty())
+ .file_size_in_bytes(1).build().unwrap()
+ };
+ let pos_v2 = file(DataContentType::PositionDeletes, DataFileFormat::Parquet);
+ let eq = file(DataContentType::EqualityDeletes, DataFileFormat::Parquet);
+ let dv = file(DataContentType::PositionDeletes, DataFileFormat::Puffin);
+ assert!(is_deletion_vector(&dv));
+ assert!(!is_deletion_vector(&pos_v2) && !is_deletion_vector(&eq));
+ assert!(v2_delete_file_error(&eq).to_string().contains("equality delete file ('d')"));
+ assert!(v2_delete_file_error(&pos_v2).to_string().contains("V2 position delete file ('d')"));
+ }Pitfall (medium, flakiness): at test time, the e2e fixture downloads pyspark from PyPI and the Iceberg jars from Maven Central. The 3× retry helps, but the jobs still depend on the network. Consider caching the uv and ivy directories in |
01720d9 to
3d00f4a
Compare
|
I asked the SI whether the previously found issues are fixed. Lines 2 and 4 below should be easy fixes. |
75d0c64 to
9291b9f
Compare
48f1c8d to
2e3a140
Compare
|
@ryzhyk for 1, I think based on what i evaluated, it should be follow-up instead of making this big PR even bigger for 2 i initially thought let's not categorize errors and it is safe to keep retrying if we're marking connector unhealthy. but now it is fixed, same for 4, fixed it! |
|
if u don't mind bigger PR to review, i can add 1 in this PR as well :P |
|
A separate PR is fine by me. |
|
Since you asked for a detailed review, I've got one for you. |
2e3a140 to
9350f8a
Compare
|
@ryzhyk thanks! fixed the findings! 🫡 |
| deletion_vectors: live_dvs, | ||
| v2_delete_file, | ||
| } = self | ||
| .live_deletes(table, &manifest_list, &unpaired_removed, find_v2) |
There was a problem hiding this comment.
perf, medium: unpaired_removed is non-empty for every snapshot that removes a data file which never had a DV (a plain copy-on-write DELETE/overwrite, the common case after 1.10), so each such snapshot re-reads every live delete manifest of the table. On a large merge-on-read table with many delete manifests, follow cost becomes proportional to table size per snapshot, not to the change. Cheap mitigations: skip when the snapshot summary has total-delete-files == 0, prune delete manifests by partition summary of the removed files, or cache the live-DV index across snapshots (it only changes by each snapshot's added/removed DVs).
| // Delta Spark fixtures. | ||
| let mut attempt = 1; | ||
| let output = loop { | ||
| let output = std::process::Command::new("uv") |
There was a problem hiding this comment.
flaky-risk, medium: each case downloads a pyspark wheel from PyPI (uv run --with pyspark==…) and the Iceberg runtime/bundle jars from Maven Central at test time, and cases run as parallel processes sharing ~/.ivy2. That is an unmocked network dependency in the adapters CI job; the 3× retry papers over it rather than removing it. Consider pre-fetching pyspark (e.g. requirements.ci.txt) and the jars in a CI step (or giving each process its own spark.jars.ivy) so the test itself never hits the network.
|
What I ran
Gates: tests, docs, changelog, breaking-change note and licences (roaring, opendal and crc32fast are MIT/Apache, and opendal uses Issues (see the inline comments)
Gaps in QA coverage
|
9350f8a to
9d60f1a
Compare
ryzhyk
left a comment
There was a problem hiding this comment.
No more bugs found. There's a performance issue for you to assess. Sounds like it's time to merge this.
New findings
- Medium (cost, not correctness): the leftover lookup rereads every delete manifest of the table.
- When: every snapshot that removes a data file without its deletion vector triggers it. That covers copy-on-write DML, INSERT OVERWRITE and partition deletes on any table that has deletion vectors.
- Wasted on new writers: this includes tables written only by Iceberg 1.10+, where the lookup can never find anything, because 1.10 removes a file's vector in the same commit.
- Where: IcebergInputEndpointInner::live_deletes (input.rs:2058), called from snapshot_delta (:1932-1947). iceberg-rust's ManifestReader::read has no cache.
- Observed: in the reviewer's trace, which I checked, a table with 39 delete manifests and 4 followed copy-on-write deletes fetched each delete manifest 5 times: once for the initial read and once per snapshot.
- Cheap fix: Java writes its version into every snapshot summary; our test table shows iceberg-version: "Apache Iceberg 1.10.0 (commit …)". Skip the lookup for Java 1.10+ writers. Pruning manifests by partition, or caching them per poll, would also help.
- Wrong comment: the comment in IcebergInputEndpointInner::live_data_files (:2113-2116) says this check "reads no manifest". In the reviewer's table history, a 1.9.2 partition delete and a 1.10 INSERT OVERWRITE each dropped a leftover vector while an older data file stayed in the table, so the method did read manifests. The code handles this correctly; only the comment is wrong.
- Low (test gap, confirmed by mutation): nothing tests whether a vector's data file is still in the table.
- The check: when a snapshot removes a vector on its own, snapshot_delta asks live_data_files whether the vector's data file is still in the table. If it is, the connector stops with a spec-violation error.
- Mutation: I made live_data_files report every data file as gone. All three Spark e2e tests still passed. No unit test reaches snapshot_delta; that code isn't even built into the unit-test binary.
- Risk: a regression here would silently ignore a vector removed from a file still in the table, so rows the table now has again would stay deleted in Feldera.
- Fix: move the check into a plain function with unit tests, or add a 1.10 INSERT OVERWRITE to the test table that drops a leftover vector while an older file stays.
- Minor: both deletion-vector e2e tests start before the first vector, so the initial table read never meets one.
- Low (observed): on local tables, a temporary I/O error now stops the connector at once.
- Cause: local tables are read through FileIO::new_with_fs(). That storage reports every I/O error as DataInvalid, keeping only the message text, and input::is_permanent_read_error treats DataInvalid as permanent.
- Observed: with all file descriptors in use, a vector read gave up after one attempt in 1 ms, with Too many open files (os error 24).
- Before this PR: these errors were retried, in snapshot mode too.
- Fix: in the iceberg-rust fork, keep the original I/O error attached and classify by its kind.
- Low (inferred, not run): retries can't recover from an error that needs a fresh table.
- What changed: manifest and vector reads now retry; before, one failure was fatal. The retries reuse the table opened for the current poll, and max_retries has no limit by default.
- Scenario: a REST catalog hands out short-lived S3 credentials, and a catch-up runs past their expiry. The connector would then retry forever instead of failing. Data-file reads already behaved this way.
- Suggestion: handle this in the retry follow-up PR (item a).
- Low (inferred, not run): stopping at once on a missing file is new for most reads. It could misfire where S3 replication or Multi-Region Access Points briefly return "not found" for objects that exist. That is a rare setup.
- Low (observed): the changelog entries land in a released section. After merging into current main, the three entries end up under ## v0.377.0 instead of ## Unreleased.
9d60f1a to
a614c9a
Compare
| .output() | ||
| .unwrap_or_else(|e| panic!("failed to run '{script}' (is `uv` on PATH?): {e}")) | ||
| }; | ||
| // Jar downloads from Maven Central fail now and then; retry like the |
There was a problem hiding this comment.
pitfall (medium): flakiness. Every CI run of these four tests resolves PyPI (pyspark 4.0.1 and 3.5.7) and Maven Central (Spark 4/3.5 Iceberg runtimes plus AWS bundles), and only three attempts with 5 s and 10 s back-off protect it. A Maven or PyPI outage longer than about 15 s turns test-adapters red for reasons the PR does not control. Could CI cache the Ivy dir (~/.ivy2) and the uv cache (actions/cache keyed on the script hash)? Or could the fixture tables be pre-built in a pinned image, as iceberg-rest already is?
|
Review: Iceberg follow mode reads deletion vectors What I ran
Gates: tests ✅, docs and changelog ✅, breaking change declared ✅, new deps Correctness checks: I checked that a transient S3 or Parquet error does not become permanent. iceberg-rust maps Missing test (medium): no test covers Flakiness (medium): see the inline comment about the Maven/PyPI downloads in the e2e fixture. |
Follow mode now reads V3 deletion vectors, so merge-on-read DELETE, UPDATE and MERGE stream as changes. For a new vector it reads the data file and keeps only the newly deleted rows. It also applies the vectors that writers before Iceberg 1.10 leave in the table. V2 delete files still stop the connector with an error. A summary of each delete manifest is kept, so a snapshot reads only new delete manifests. Manifest and deletion vector reads now retry like data-file reads, and each retry marks the connector unhealthy. A read that a retry cannot fix, such as a read of a missing file, stops at once. Signed-off-by: Swanand Mulay <73115739+swanandx@users.noreply.github.com>
b3aec76 to
859afe7
Compare
|
Review: LGTM, with one gap to note. What I ran
I also checked that the tech-debt (medium): the new e2e tests don't run in CI. No workflow enables Cases the PR's tests don't cover
|
Follow mode now reads Iceberg V3 deletion vectors (DVs), so merge-on-read
DELETE,UPDATEandMERGEstream as changes. Before, the connector stopped with an error.Writers before Iceberg 1.10 leave the DV of a removed file in the table. The connector applies that DV to the removed file, as the Java reader does, and ignores it after the file is gone.
Cost: for a new DV, the connector reads the whole data file once and keeps only the newly deleted rows. A copy-on-write change reads two files: the old file and its rewrite. On MinIO, a
DELETEof 1% of 10M rows in 100 files took 4.5 s and 113 MB with DVs, and 19.6 s and 2.0 GB with copy-on-write. The connector keeps a small summary of each delete manifest that it read, so a snapshot reads only new delete manifests and those that name a removed file. When a snapshot drops a leftover DV, the connector reads the data manifests old enough to hold its data file, to make sure that the file is gone.Manifest and deletion vector reads now retry like data-file reads, and each retry marks the connector unhealthy. A read that a retry cannot fix, such as a read of a missing file in follow mode, stops at once. On a table on the local disk, an invalid-data error still retries, because iceberg-rust's local reader reports a temporary I/O error the same way.
Linked Issue
Part of #6165.
Describe Manual Test Plan
A Spark fixture writes a partitioned V3 table, whose partition spec changes after the first insert, with merge-on-read
DELETE,UPDATEandMERGE, a copy-on-writeDELETEand a compaction. Each data file has many row groups. A second fixture writes with Iceberg 1.9.2 and then 1.10.0, so the table has a leftover DV. It also writes a V2 table with a position delete file. The e2e tests check the final rows and the number of rows read. If any branch above breaks, a test fails. A third fixture writes every type that Spark can write, then deletes and updates rows. That test checks that follow mode gives the same rows as a snapshot read of the last snapshot. Spark cannot writetimeorfixed, and a unit test runs the row filter on every type family.Checklist
Breaking Changes?
Mark if you think the answer is yes for any of these components:
Describe Incompatible Changes
Follow mode now stops with an error when a snapshot deletes rows and the table still has a V2 delete file anywhere. Before, it went on, and emitted rows that were already deleted whenever that delete file applied to a removed file.