Repository navigation
feat: Add PolarisSource and PolarisProvider for Apache Polaris - #7006
Merged
Merged
Conversation
Add Polaris support on top of the Iceberg REST catalog integration: - PolarisSource(IcebergSource): endpoint defaults to $POLARIS_HOST/api/catalog, OAuth2 client-credentials or bearer-token auth read from env vars (only names are stored in the registry), credential vending, and a discover() helper that builds sources for tables in a catalog. - PolarisClient: OAuth client-credentials token acquisition with refresh, and table property updates through the standard Iceberg REST updateTable (set-properties / remove-properties). - PolarisProvider: on feast apply, record feast.* table properties for feature views backed by a PolarisSource; deleted views are marked, tables are kept. Registered as provider "polaris". - Centralize the PyIceberg REST catalog config in IcebergSource.get_pyiceberg_catalog_config()/load_pyiceberg_table() and use it from the DuckDB and Spark offline stores, replacing three duplicated copies. Behavior for existing IcebergSource users is unchanged. Refs feast-dev#7005 Signed-off-by: hao-xu5 <hxu44@apple.com>
| "No Polaris credentials found: set %s and %s (OAuth client " | ||
| "credentials) or %s (bearer token)", | ||
| self.client_id_env_var, | ||
| self.client_secret_env_var, |
|
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #7006 +/- ##
==========================================
- Coverage 51.02% 50.94% -0.09%
==========================================
Files 452 455 +3
Lines 56728 56915 +187
Branches 8265 8290 +25
==========================================
+ Hits 28946 28994 +48
- Misses 25801 25940 +139
Partials 1981 1981
Continue to review full report in Codecov by Harness.
🚀 New features to boost your workflow:
|
Collaborator
Author
|
@ntkathole PTAL, thanks! |
Allowlist the env var names used as test fixtures (detect-secrets false positives) and stop logging credential env var names (CodeQL py/clear-text-logging-sensitive-data). Signed-off-by: hao-xu5 <hxu44@apple.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this PR does / why we need it:
Adds Apache Polaris support on top of the Iceberg REST catalog integration (
iceberg_catalogcontrib), mirroring the Unity Catalog layering from #6499.PolarisSource(IcebergSource):endpointdefaults to$POLARIS_HOST/api/catalog; OAuth2 client-credentials or bearer-token auth read from env vars (only the variable names are stored in the registry); credential vending for reads/writes;PolarisSource.discover()builds sources for tables in a catalog.PolarisClient: OAuth client-credentials token acquisition with refresh before expiry, and table property updates via the standard Iceberg RESTupdateTable(set-properties/remove-properties).PolarisProvider(provider: polaris): onfeast apply, recordsfeast.project,feast.feature_view,feast.owner,feast.description,feast.tag.*andfeast.is_feature_tableon the backing Polaris tables. A deleted feature view keeps the table and marks itfeast.is_feature_table=false+feast.deleted_at. A failed sync is logged and never failsfeast apply.IcebergSource.get_pyiceberg_catalog_config()/load_pyiceberg_table()so subclasses can override it. Behavior for existingIcebergSourceusers is unchanged (covered by a test).Polaris has no lineage API, so there is no
sync_lineageequivalent.Not included (follow-up): column projection / predicate pushdown on the read path, which needs changes to how the offline stores call into the source. Multi-level namespaces keep the existing REST client behavior (single-level only).
Which issue(s) this PR fixes:
Refs #7005 (related: #6499)
Checks
git commit -s)Testing Strategy
Unit: 33 new tests in
iceberg_catalog/tests/test_polaris.py(HTTP mocked); ruff and mypy clean. In the existingiceberg_catalogtests, 7 fail identically onmasterand with this change: 2 DuckDB tests that patchget_catalog_clienton a REST source, and 5 Spark tests that needpyspark.Manual: ran against a local Apache Polaris (
apache/polaris:latest) with MinIO: table discovery, DuckDB read with OAuth + vended credentials,feast applywithprovider: polaris(properties verified by reloading the table through PyIceberg),get_historical_features,materialize+ online read, feature view deletion, static-token mode, and bad credentials (sync returns False, no exception).Misc
Docs are updated in
docs/reference/data-sources/iceberg.md.