Repository navigation
[feldera-types] Accept az:// buckets for checkpoint sync - #7972
Merged
Merged
Conversation
Contributor
|
What I ran
Gates: tests, docs, changelog and commit message are all in place. There is no QA gaps (medium):
Otherwise LGTM. |
bestouff
force-pushed
the
azure-blob-checkpoint-sync
branch
2 times, most recently
from
November 13, 2026 15:49
c4989bc to
e94ed56
Compare
gz
approved these changes
Nov 13, 2026
Checkpoint sync could reach Azure Blob Storage only through a gateway that translates S3 calls, which the user must deploy and maintain. A sync `bucket` that starts with `az://` now selects Azure Blob Storage, as `gs://` selects Google Cloud Storage. `access_key` is the name of the storage account and `secret_key` its shared key. Without a shared key, rclone reads the credentials from the environment, for example an AKS Workload Identity. `validate` rejects a `read_bucket` whose scheme differs from that of `bucket`, for the three object stores, an `az://` bucket without the account name, and an `az://` bucket without a container name. rclone builds the service URL from the account name, and no credential in the environment supplies it. Only a start in standby mode called `validate`, so a pipeline with a bad sync config started and failed later with an rclone error. `validate_config` now calls it, so each start does the checks. This is an incompatible change: a config with `read_bucket` equal to `bucket`, with the two in different object stores, or with the deprecated `standby` field set, does not start anymore. The changelog and the checkpoint sync page say so. The enterprise sync-checkpoint crate implements the transfer. Signed-off-by: Xavier Bestel <xavier.bestel@feldera.com>
bestouff
force-pushed
the
azure-blob-checkpoint-sync
branch
from
November 13, 2026 16:40
e94ed56 to
89291b6
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Checkpoint sync can reach Azure Blob Storage only through a gateway that translates S3 calls, which the user must deploy and maintain.
A sync
bucketthat starts withaz://now selects Azure Blob Storage, asgs://selects Google Cloud Storage. The enterprise sync-checkpoint crate implements the transfer (feldera/cloud#2220). This PR has the config validation, the docs and the changelog.syncfieldaz://bucketaz://<container>[/<path>]. The container must exist.access_keysecret_keyendpointprovider,regionSyncConfig::validatenow rejects:read_bucketwhose object store differs from that ofbucket, for the three stores (before:gs://against S3 only);az://bucket withoutaccess_key, because rclone builds the service URL from the account name;az://bucket without a container name.Only a start in standby mode called
validate(crates/connectors/src/controller/sync.rs:238), so on a usual start these checks did nothing.validate_config(crates/connectors/src/controller/validate.rs) now calls it, so each start does the checks, and a failure is the newConfigError::InvalidSyncConfig. The docs have a new section "Configuration checks at start".openapi.jsonandtypes.gen.tsare regenerated; only doc comments change.Describe Manual Test Plan
cargo test -p feldera-types sync_config_tests: 6 passed, andcargo test -p dbsp_connectors --lib bad_sync_config_fails_validationpasses. With the call fromvalidate_configor the container check disabled, the related test fails. With theaz://detection disabled, the new assertions fail.Storage Blob Data Contributor,Storage Blob Data Reader) come from the Azure documentation, not from a test.Checklist
Breaking Changes?
Yes, in the changelog as an incompatible change. Each start now does the sync config checks. These configs started before on a usual start and do not start now:
read_bucketequal tobucketread_bucketandbucketin different object storesstandby: trueinitial=standbyMerge feldera/cloud#2220 first: this PR documents the feature.