Repository navigation
[connectors] Delta: report the follow phase after a resume past the snapshot - #7797
Merged
Merged
Conversation
…napshot. A snapshot_and_follow connector that resumed from a checkpoint taken after its snapshot completed kept its initial phase, so input_connector_delta_phase read 0 (loading_snapshot) while the connector followed the log. Set the phase to follow in that case. The snapshot completion time and record count are not in the checkpoint, so document that they cover only the current run. Fixes #7747. Signed-off-by: Ben Pfaff <blp@feldera.com>
The snapshot completion time and record count are not in the checkpoint. After a resume past the snapshot they read 0, but their help text said that 0 means the snapshot is not complete. The phase metric is correct in that case, because the follow loop sets it. Refs #7747. Signed-off-by: Ben Pfaff <blp@feldera.com>
Contributor
|
LGTM.
The new wait polls with a 20 s timeout and has no fixed sleep, so it should not make the test flaky. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Linked Issue
Closes #7747.
A
snapshot_and_followDelta connector that resumed from a checkpoint taken after its snapshot completed reportedinput_connector_delta_phase = 0(loading_snapshot) while it followed the log.crates/connectors/src/integrated/delta_table/input.rs). The snapshot completion time and record count are not in the checkpoint, so their help text now says that they cover only the current run.follow_loopsets the phase on every path that follows. Its two snapshot metrics have the same limit as Delta, so their help text gets the same change.docs.feldera.com/docs/operations/metrics.mdis generated from/metricsoutput. I edited only the four changed rows, to the new help strings.Describe Manual Test Plan
test_follownow waits for phase 1 after each resume. This covers thesnapshot_and_follow_file_test_suspendtests instatic-connectors, which run without transactions and with thesnapshotandalwaystransaction modes.snapshot_and_followvariants fail:timeout waiting for connector phase 1 (current: 0)follow_file_test_suspendtests pass, two runsThe Iceberg change is text only. I did not run the Iceberg follow tests, which need MinIO and a REST catalog.
Checklist
Breaking Changes?
Mark if you think the answer is yes for any of these components:
Describe Incompatible Changes
None.