Repository navigation
[iceberg] Skip already queued rows on a follow-mode retry - #7869
Conversation
|
LGTM. What I ran
I checked the per-file skip-by-count against the parallel reader (per-file order holds with Coverage gap (medium): the integration test breaks a whole file, so on retry each queued file is skipped in full. Two cases only have unit coverage of |
bestouff
left a comment
There was a problem hiding this comment.
Some smallish changes, but otherwise LGTM
A follow-mode read that failed part way was retried from the start, so its rows were ingested twice. A retry now skips, in each data file, the rows that earlier attempts queued. Signed-off-by: Swanand Mulay <73115739+swanandx@users.noreply.github.com>
458e7e9 to
bebbf98
Compare
Linked Issue
Follow-up to #7590.
In follow mode, a read that failed part way was retried from the start, so its rows were ingested twice. Now the retry skips the rows that earlier attempts already queued, for each data file.
A restart from a checkpoint taken during a retry still reads the snapshot again (at-least-once), as the docs already say.
Describe Manual Test Plan
New test
rest_follow_retry_does_not_repeat_rows: it breaks the last data file of a snapshot, waits for the read to fail, restores the file, and checks that each row is ingested once. It passed 5 of 5 runs. Without the skip, the test fails with rows ingested twice. The full Iceberg suite passes.Checklist
Breaking Changes?
Mark if you think the answer is yes for any of these components:
Describe Incompatible Changes
None.