← 输入框|github.com|刷新
Skip to content

[connectors] Delta: report the follow phase after a resume past the snapshot - #7797

Merged
blp merged 2 commits into
mainfrom
delta-resume-phase
Nov 6, 2026
Merged

blp merged 2 commits into
mainfrom
delta-resume-phase

Conversation

@blp

@blp blp commented Nov 3, 2026

Copy link
Copy Markdown
Member

Linked Issue

Closes #7747.

A snapshot_and_follow Delta connector that resumed from a checkpoint taken after its snapshot completed reported input_connector_delta_phase = 0 (loading_snapshot) while it followed the log.

Commit Change
Delta Set the phase to follow when the connector resumes past the snapshot (crates/connectors/src/integrated/delta_table/input.rs). The snapshot completion time and record count are not in the checkpoint, so their help text now says that they cover only the current run.
Iceberg No phase bug: follow_loop sets the phase on every path that follows. Its two snapshot metrics have the same limit as Delta, so their help text gets the same change.

docs.feldera.com/docs/operations/metrics.md is generated from /metrics output. I edited only the four changed rows, to the new help strings.

Describe Manual Test Plan

test_follow now waits for phase 1 after each resume. This covers the snapshot_and_follow_file_test_suspend tests in static-connectors, which run without transactions and with the snapshot and always transaction modes.

Run Result
Without the fix 3 snapshot_and_follow variants fail: timeout waiting for connector phase 1 (current: 0)
With the fix 5/5 follow_file_test_suspend tests pass, two runs

The Iceberg change is text only. I did not run the Iceberg follow tests, which need MinIO and a REST catalog.

Checklist

  • Unit tests added/updated
  • Integration tests added/updated
  • Documentation updated
  • Changelog updated

Breaking Changes?

Mark if you think the answer is yes for any of these components:

Describe Incompatible Changes

None.

blp added 2 commits November 2, 2026 16:35
…napshot.

A snapshot_and_follow connector that resumed from a checkpoint taken
after its snapshot completed kept its initial phase, so
input_connector_delta_phase read 0 (loading_snapshot) while the
connector followed the log.

Set the phase to follow in that case. The snapshot completion time and
record count are not in the checkpoint, so document that they cover
only the current run.

Fixes #7747.

Signed-off-by: Ben Pfaff <blp@feldera.com>
The snapshot completion time and record count are not in the
checkpoint. After a resume past the snapshot they read 0, but their
help text said that 0 means the snapshot is not complete. The phase
metric is correct in that case, because the follow loop sets it.

Refs #7747.

Signed-off-by: Ben Pfaff <blp@feldera.com>
@mythical-fred-oss

Copy link
Copy Markdown
Contributor

LGTM.

Ran Result
cargo test -p static-connectors --test connector_suites follow_file_test_suspend 5/5 pass
Same test with the else branch in input.rs reverted snapshot_and_follow_file_test_suspend fails: timeout waiting for connector phase 1 (current: 0). The new assertion catches the regression.
pre-commit run on the changed files pass

The new wait polls with a 20 s timeout and has no fixed sleep, so it should not make the test flaky.

@blp
blp requested a review from anandbraman November 3, 2026 03:00
@blp blp self-assigned this Nov 3, 2026
@blp blp added ft Fault tolerant, distributed, and scale-out implementation connectors Issues related to the adapters/connectors crate QA Testing and quality assurance rust Pull requests that update Rust code enterprise Issue related to Feldera Enterprise features. labels Nov 3, 2026

@anandbraman anandbraman left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm, thanks!

@blp
blp added this pull request to the merge queue Nov 6, 2026
Merged via the queue into main with commit edafe9e Nov 6, 2026
3 checks passed
@blp
blp deleted the delta-resume-phase branch November 6, 2026 19:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

connectors Issues related to the adapters/connectors crate enterprise Issue related to Feldera Enterprise features. ft Fault tolerant, distributed, and scale-out implementation QA Testing and quality assurance rust Pull requests that update Rust code

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[connectors] Delta: snapshot metrics read "loading snapshot" after resuming from a checkpoint

2 participants