← 输入框|github.com|刷新
Skip to content

[feldera-types] Accept az:// buckets for checkpoint sync - #7972

Merged
bestouff merged 1 commit into
mainfrom
azure-blob-checkpoint-sync
Nov 13, 2026
Merged

bestouff merged 1 commit into
mainfrom
azure-blob-checkpoint-sync

Conversation

@bestouff

@bestouff bestouff commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Checkpoint sync can reach Azure Blob Storage only through a gateway that translates S3 calls, which the user must deploy and maintain.

A sync bucket that starts with az:// now selects Azure Blob Storage, as gs:// selects Google Cloud Storage. The enterprise sync-checkpoint crate implements the transfer (feldera/cloud#2220). This PR has the config validation, the docs and the changelog.

sync field Meaning for az://
bucket az://<container>[/<path>]. The container must exist.
access_key Name of the storage account. Required.
secret_key Shared key of the account. If absent, rclone reads the credentials from the environment (AKS Workload Identity, managed identity).
endpoint Blob service endpoint. Only for an emulator or a sovereign cloud.
provider, region Ignored

SyncConfig::validate now rejects:

  • a read_bucket whose object store differs from that of bucket, for the three stores (before: gs:// against S3 only);
  • an az:// bucket without access_key, because rclone builds the service URL from the account name;
  • an az:// bucket without a container name.

Only a start in standby mode called validate (crates/connectors/src/controller/sync.rs:238), so on a usual start these checks did nothing. validate_config (crates/connectors/src/controller/validate.rs) now calls it, so each start does the checks, and a failure is the new ConfigError::InvalidSyncConfig. The docs have a new section "Configuration checks at start".

openapi.json and types.gen.ts are regenerated; only doc comments change.

Describe Manual Test Plan

  • cargo test -p feldera-types sync_config_tests: 6 passed, and cargo test -p dbsp_connectors --lib bad_sync_config_fails_validation passes. With the call from validate_config or the container check disabled, the related test fails. With the az:// detection disabled, the new assertions fail.
  • The transfer itself is tested in feldera/cloud#2220 against the Azurite emulator: push, pull, garbage collection, bucket ownership, solo and multihost.
  • Not tested: authentication from the environment on AKS, and a real Azure storage account. The roles in the docs (Storage Blob Data Contributor, Storage Blob Data Reader) come from the Azure documentation, not from a test.
  • I did not build the docs site.

Checklist

  • Unit tests added/updated
  • Integration tests added/updated (in feldera/cloud#2220)
  • Documentation updated
  • Changelog updated

Breaking Changes?

Yes, in the changelog as an incompatible change. Each start now does the sync config checks. These configs started before on a usual start and do not start now:

Config Before Now
read_bucket equal to bucket Started; the fallback was of no use Does not start
read_bucket and bucket in different object stores Started; the fallback pull went to the wrong service Does not start
Deprecated standby: true Started as a usual pipeline; the field was ignored Does not start; the error says to use initial=standby

Merge feldera/cloud#2220 first: this PR documents the feature.

Comment thread crates/feldera-types/src/config.rs Resolved
@mythical-fred-oss

Copy link
Copy Markdown
Contributor

What I ran

Check Result
cargo test --locked -p feldera-types sync_config ✅ 5 passed
pre-commit run --files <changed files> (clippy, fmt, machete, OpenAPI/client regen, cargo doc) ✅ all passed

Gates: tests, docs, changelog and commit message are all in place. There is no unsafe, no new dependency and no CI change. The change isn't breaking: a bucket starting with az:// was never a valid S3 bucket name.

QA gaps (medium):

  • An empty Azure container (az://, az:///dir) passes validation (see inline comment).
  • Nothing in this repo covers the Azure transfer path. python/tests/utils.py only switches between S3 and gs, so the platform checkpoint-sync tests can't exercise az://. Consider adding an az scheme there, backed by Azurite, so the OSS suite covers it too.

Otherwise LGTM.

@bestouff
bestouff requested a review from gz November 13, 2026 12:26
@bestouff
bestouff force-pushed the azure-blob-checkpoint-sync branch 2 times, most recently from c4989bc to e94ed56 Compare November 13, 2026 15:49
Checkpoint sync could reach Azure Blob Storage only through a gateway
that translates S3 calls, which the user must deploy and maintain.

A sync `bucket` that starts with `az://` now selects Azure Blob Storage,
as `gs://` selects Google Cloud Storage. `access_key` is the name of the
storage account and `secret_key` its shared key. Without a shared key,
rclone reads the credentials from the environment, for example an AKS
Workload Identity.

`validate` rejects a `read_bucket` whose scheme differs from that of
`bucket`, for the three object stores, an `az://` bucket without the
account name, and an `az://` bucket without a container name. rclone
builds the service URL from the account name, and no credential in the
environment supplies it.

Only a start in standby mode called `validate`, so a pipeline with a
bad sync config started and failed later with an rclone error.
`validate_config` now calls it, so each start does the checks. This is
an incompatible change: a config with `read_bucket` equal to `bucket`,
with the two in different object stores, or with the deprecated
`standby` field set, does not start anymore. The changelog and the
checkpoint sync page say so.

The enterprise sync-checkpoint crate implements the transfer.

Signed-off-by: Xavier Bestel <xavier.bestel@feldera.com>
@bestouff
bestouff force-pushed the azure-blob-checkpoint-sync branch from e94ed56 to 89291b6 Compare November 13, 2026 16:40
@bestouff
bestouff added this pull request to the merge queue Nov 13, 2026
Merged via the queue into main with commit 0f6842d Nov 13, 2026
1 check passed
@bestouff
bestouff deleted the azure-blob-checkpoint-sync branch November 13, 2026 17:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants