feat: Support table format: Iceberg, Delta, and Hudi by haoxu0 · Pull Request #5650 · feast-dev/feast

haoxu0 · 2025-10-05T21:47:54Z

What this PR does / why we need it:

examples:

  from feast.table_format import IcebergFormat
  from feast.infra.offline_stores.contrib.spark_offline_store.spark_source import SparkSource

  # Create Iceberg format with snapshot option
  iceberg_format = IcebergFormat()
  iceberg_format.set_property("snapshot-id", "your_snapshot_id")

  spark_source = SparkSource(
      name="iceberg_features",
      path="my_catalog.my_database.my_table",  # Catalog table name
      table_format=iceberg_format
  )

  Generated Spark code:
  spark.read.format("iceberg").option("snapshot-id", "your_snapshot_id").load("my_catalog.my_database.my_table")

  ✅ Delta Example:

  from feast.table_format import DeltaFormat

  delta_format = DeltaFormat()
  delta_format.set_property("versionAsOf", "1")

  spark_source = SparkSource(
      name="delta_features",
      path="s3://bucket/delta-table",
      table_format=delta_format
  )

  Generated Spark code:
  spark.read.format("delta").option("versionAsOf", "1").load("s3://bucket/delta-table")

  ✅ Hudi Example:

  from feast.table_format import HudiFormat

  hudi_format = HudiFormat()
  hudi_format.set_property("hoodie.datasource.query.type", "snapshot")
  hudi_format.set_property("as.of.instant", "20231010120000")

  spark_source = SparkSource(
      name="hudi_features",
      path="s3://bucket/hudi-table",
      table_format=hudi_format
  )

  Generated Spark code:
  spark.read.format("hudi")\
    .option("hoodie.datasource.query.type", "snapshot")\
    .option("as.of.instant", "20231010120000")\
    .load("s3://bucket/hudi-table")

Which issue(s) this PR fixes:

Misc

Signed-off-by: HaoXuAI <sduxuhao@gmail.com>

haoxu0 · 2025-10-05T22:29:38Z

    string date_partition_column_format = 5;
+
+    // Table Format (e.g. iceberg, delta, etc)
+    string table_format = 6;


TODO, create TableFormat proto, consolidate with FileFormat proto

tokoko · 2025-10-06T09:58:50Z

+1 on the inclusion of all 3 formats. Still I think we might be able to better design data-source side such that data source definitions don't tie the sources to specific offline stores. For example right now FileSource already supports specifying delta as a format... I don't like that design either because delta and especially iceberg might require more configuration that plain files, but the upside in that design was that Source class wasn't tied to a single offline store and it is actually usable from both dask and duckdb offline stores. Hiding formats behind SparkSource will mean that when a user stores a data source instance in the registry only spark offline store can be used to retrieve the data.

I think we can have best of both worlds if we instead go for adding all these formats as separate independent data sources (DeltaSource, IcebergSource, HudiSource) and then simply teach spark engine (and in the future others) how to read it.

franciscojavierarceo · 2025-10-06T12:21:05Z

            query: The query to be executed in Spark.
            path: The path to file data.
-            file_format: The format of the file data.
+            file_format: The underlying file format (parquet, avro, csv, json).


why not consolidate now?

franciscojavierarceo · 2025-10-06T12:22:28Z

I think we can have best of both worlds if we instead go for adding all these formats as separate independent data sources (DeltaSource, IcebergSource, HudiSource) and then simply teach spark engine (and in the future others) how to read it.

+1

haoxu0 · 2025-10-06T22:58:09Z

@franciscojavierarceo @tokoko consolidation with FilleFormat and new data sources could break the backward compatibility, so I want to do it pace by pace.

franciscojavierarceo · 2025-10-07T00:02:09Z

That makes sense

tokoko · 2025-10-07T08:46:29Z

@HaoXuAI Why would new data sources break backwards compatibility though?

haoxu0 · 2025-10-09T05:01:50Z

@HaoXuAI Why would new data sources break backwards compatibility though?

There will be some proto changes, no 100% sure if there will be API changes exposed to users but I think might be the case

Signed-off-by: HaoXuAI <sduxuhao@gmail.com>

haoxu0 · 2025-10-10T05:11:42Z

@franciscojavierarceo @ntkathole mind take a look

franciscojavierarceo

@HaoXuAI i don't see use actually using or testing Spark Table, Iceberg, or Hudi format's outside of our definitions, can you add that?

Can you also add documentation that these formats are now supported?

Otherwise lgtm.

haoxu0 · 2025-10-14T21:20:25Z

@HaoXuAI i don't see use actually using or testing Spark Table, Iceberg, or Hudi format's outside of our definitions, can you add that?

Can you also add documentation that these formats are now supported?

Otherwise lgtm.

haoxu0 · 2025-10-14T21:21:36Z

@HaoXuAI i don't see use actually using or testing Spark Table, Iceberg, or Hudi format's outside of our definitions, can you add that?
Can you also add documentation that these formats are now supported?
Otherwise lgtm.

Gonna update to add the TableFormat proto in the next PR, after that I'll add the docs. And I think the test will need to be changed as well.

Signed-off-by: hao-xu5 <hxu44@apple.com>

haoxu0 · 2025-11-06T01:49:49Z

@franciscojavierarceo mind take another look?

franciscojavierarceo

Lgtm

* add support for table format such as Iceberg, Delta, Hudi etc. Signed-off-by: HaoXuAI <sduxuhao@gmail.com> * linting Signed-off-by: HaoXuAI <sduxuhao@gmail.com> * linting Signed-off-by: HaoXuAI <sduxuhao@gmail.com> * add tests Signed-off-by: HaoXuAI <sduxuhao@gmail.com> * fix tests Signed-off-by: HaoXuAI <sduxuhao@gmail.com> * fix tests Signed-off-by: HaoXuAI <sduxuhao@gmail.com> * linting Signed-off-by: HaoXuAI <sduxuhao@gmail.com> * add tableformat proto Signed-off-by: hao-xu5 <hxu44@apple.com> * update Signed-off-by: hao-xu5 <hxu44@apple.com> * update doc Signed-off-by: hao-xu5 <hxu44@apple.com> * fix linting Signed-off-by: hao-xu5 <hxu44@apple.com> * fix test Signed-off-by: hao-xu5 <hxu44@apple.com> --------- Signed-off-by: HaoXuAI <sduxuhao@gmail.com> Signed-off-by: hao-xu5 <hxu44@apple.com> Co-authored-by: hao-xu5 <hxu44@apple.com>

# [0.57.0](v0.56.0...v0.57.0) (2025-11-13) ### Bug Fixes * Improve trino to feast type mapping with (real,varchar,timestamp,decimal) ([#5691](#5691)) ([f855ad2](f855ad2)) * Materialize API - ODFV views not looked-up (thinks views non existant) - crashes materialize ([#5716](#5716)) ([1b050b3](1b050b3)) * Support historical feature retrieval with start_date/end_date in RemoteOfflineStore ([#5703](#5703)) ([ad32756](ad32756)) * Thread safe Clickhouse offline store ([#5710](#5710)) ([5f446ed](5f446ed)) ### Features * Add annotations to cronjob CRDs ([#5701](#5701)) ([be6e6c2](be6e6c2)) * Add batch commit mode for MySQL OnlineStore ([#5699](#5699)) ([3cfe4eb](3cfe4eb)) * Add possibility to materialize only latest values, to increase performance ([#5713](#5713)) ([8d77b72](8d77b72)) * Support table format: Iceberg, Delta, and Hudi ([#5650](#5650)) ([2915ad1](2915ad1))

Extends the existing TableFormat abstraction (feast-dev#5650) with Lance rather than introducing a separate data source, so Lance is addressed the same way Iceberg, Delta and Hudi already are. Closes part of feast-dev#6899. LanceFormat carries catalog/namespace addressing plus an optional pin to a dataset version or tag. Because SparkSource already drives its reader generically from table_format.format_type.value and table_format.properties, this works with SparkSource with no changes to it: format_type.value is "lance", and the pin is mirrored into properties as lance.version / lance.tag. version is validated as >= 1 so the Python and proto semantics agree. Lance dataset versions start at 1 and the proto treats 0 as unset, so without that guard version=0 would not round-trip, since to_proto/from_proto read 0 as absent. version and tag are mutually exclusive, because a tag already resolves to a version. Only DataFormat_pb2 is regenerated, using grpcio-tools 1.62.3 so the emitted gencode stays at the 4.25.1 level the other checked-in protos use. Regenerating with the pinned grpcio-tools 1.84.0 instead emits gencode that calls ValidateProtobufRuntimeVersion for protobuf 7.35.1, which would break the declared protobuf>=4.24.0 floor for that one module. DataSource_pb2 is deliberately left untouched. It is already stale against DataSource.proto on master, missing ConnectionRef entries, and regenerating it produces ~125 lines of churn unrelated to this change. Signed-off-by: hao-xu5 <hxu44@apple.com>

Extends the existing TableFormat abstraction (#5650) with Lance rather than introducing a separate data source, so Lance is addressed the same way Iceberg, Delta and Hudi already are. Closes part of #6899. LanceFormat carries catalog/namespace addressing plus an optional pin to a dataset version or tag. Because SparkSource already drives its reader generically from table_format.format_type.value and table_format.properties, this works with SparkSource with no changes to it: format_type.value is "lance", and the pin is mirrored into properties as lance.version / lance.tag. version is validated as >= 1 so the Python and proto semantics agree. Lance dataset versions start at 1 and the proto treats 0 as unset, so without that guard version=0 would not round-trip, since to_proto/from_proto read 0 as absent. version and tag are mutually exclusive, because a tag already resolves to a version. Only DataFormat_pb2 is regenerated, using grpcio-tools 1.62.3 so the emitted gencode stays at the 4.25.1 level the other checked-in protos use. Regenerating with the pinned grpcio-tools 1.84.0 instead emits gencode that calls ValidateProtobufRuntimeVersion for protobuf 7.35.1, which would break the declared protobuf>=4.24.0 floor for that one module. DataSource_pb2 is deliberately left untouched. It is already stale against DataSource.proto on master, missing ConnectionRef entries, and regenerating it produces ~125 lines of churn unrelated to this change. Signed-off-by: hao-xu5 <hxu44@apple.com>

# [0.67.0](v0.66.0...v0.67.0) (2026-10-13) ### Bug Fixes * Allow dill 0.4 so UDF serialization works on Python 3.14 ([#6858](#6858)) ([139704f](139704f)) * Allow serving features while a feature view is MATERIALIZING ([#6789](#6789)) ([e2d7fe2](e2d7fe2)), closes [#6780](#6780) * Allowlist the pinned commit SHA flagged by detect-secrets ([4b81d18](4b81d18)) * Apply point-in-time filter correctly with the Polars backend ([#6992](#6992)) ([01224d0](01224d0)) * Authorize registry server writes before deserializing UDFs ([e0ef16e](e0ef16e)) * Automate versioned documentation publication ([#6763](#6763)) ([5ad5592](5ad5592)), closes [#6762](#6762) * Avoid NumPy generic-unit deprecation when importing feast.type_map ([5fea698](5fea698)) * Bind metrics, REST registry, ui, and lineage servers dual-stack ([#6886](#6886)) ([8d2773c](8d2773c)) * Bound Arrow Flight batch size in the offline server ([#6896](#6896)) ([d169238](d169238)) * Compute std and var correctly for tiled aggregations ([213a130](213a130)) * Convert PdfBytes values to protos like ImageBytes ([#6968](#6968)) ([af20fe4](af20fe4)) * Decode fixed_size_list as the matching Feast list value type ([#6944](#6944)) ([8cfe891](8cfe891)) * Do not log /metrics requests from the pre-fork metrics server thread ([7087f1f](7087f1f)), closes [#6928](#6928) * Don't return from finally blocks in REST registry helpers ([d7cb4d0](d7cb4d0)) * Encode non-finite floats in feature server JSON responses ([#6930](#6930)) ([211ecb8](211ecb8)) * Feature freshness logic updated ([#6765](#6765)) ([f0bc070](f0bc070)) * Fix monitoring auth ([015cc84](015cc84)) * Handle non-nanosecond timestamps in tiled aggregations ([b8989cb](b8989cb)) * Improve feature list column sizing ([#6793](#6793)) ([2573693](2573693)) * Include vector attributes in Field equality ([#6934](#6934)) ([298c3f6](298c3f6)) * Increase feast container probe timeouts to avoid false restarts ([e8dbd50](e8dbd50)) * Infer the tensor type from retrieved values in to_tensor ([a03e323](a03e323)) * Infer timestamps for datetime columns in any unit or timezone ([810391f](810391f)) * Infer vector width when vector_length is undeclared ([0056565](0056565)), closes [#6907](#6907) * Initialize auth managers in standalone lineage server ([#6873](#6873)) ([ecd0d38](ecd0d38)) * Keep a zero watermark delay threshold on a Kafka source ([e2e28c7](e2e28c7)) * Keep entity rows apart in the ibis point-in-time join ([a87c070](a87c070)) * Keep the BatchFeatureView type and settings when copying it ([#6989](#6989)) ([013fc57](013fc57)) * Keep tiling, offline and stream engine settings when copying a StreamFeatureView ([fac0f88](fac0f88)) * Limit tiled max and min aggregations to the window ([2583c45](2583c45)) * Load Milvus collections once instead of on every query ([#6882](#6882)) ([f9e91fa](f9e91fa)) * Make vector length validation reachable, schema-driven and vectorized ([#6909](#6909)) ([29af8c0](29af8c0)), closes [#6907](#6907) * Map INT32_LIST, PDF_BYTES and IMAGE_BYTES type names to their ValueType ([8c1e825](8c1e825)) * Map narrow and unsigned Arrow integer types to Feast value types ([#6960](#6960)) ([ae74360](ae74360)) * Map Spark smallint and tinyint columns to INT32 ([9d42729](9d42729)) * Map SQL Server bigint to INT64 and float to DOUBLE ([6d15533](6d15533)) * Map Trino DECIMAL columns to pyarrow decimal128 ([155bc24](155bc24)) * Normalize Arrow-backed dtypes from Ray Dataset.to_pandas() for pyarrow >=25 compatibility ([#6916](#6916)) ([374f1d7](374f1d7)) * **operator:** Set grpc appProtocol on the offline Service port ([f9dcfb5](f9dcfb5)) * Pin expected tensor to same device in online retrieval test ([#6777](#6777)) ([1f2584d](1f2584d)) * Preserve an unset aggregation time window across a proto round-trip ([2000379](2000379)) * Preserve entity row order in precomputed reads ([ce37029](ce37029)) * Pull the MinIO test image from pgsty/minio ([fe27230](fe27230)) * Pull the MinIO test image from quay ([#6838](#6838)) ([367001b](367001b)) * Raise Trino monitoring query errors other than a missing table ([6161766](6161766)) * Read Kafka and Kinesis sources without a batch source from proto ([a620f5c](a620f5c)) * Read naive datetimes as UTC in UnixTimestamp values ([bc703d9](bc703d9)) * Record which storage class wrote a custom_storage payload ([#6988](#6988)) ([c0545f7](c0545f7)) * Refresh the SQL registry cache after deleting a data source ([66c55e3](66c55e3)) * Refresh the SQL registry cache after deleting a permission ([6b212f5](6b212f5)) * Reject unknown resource types and actions in the REST permission API ([a5eaa36](a5eaa36)) * Remove pip install command from Redis GLIDE docstring ([66ff990](66ff990)) * Remove runtime pip install hints from mlflow errors ([dd9f099](dd9f099)) * Remove unsupported vector_len from PostgreSQL tutorial config ([fcc9a85](fcc9a85)) * Repair Chronon operator validation and Pixi CI setup ([#6947](#6947)) ([3710f78](3710f78)) * Report Dask monitoring read errors instead of no data ([81673d9](81673d9)) * Resolve feature view batch source name against registry ([f8883d0](f8883d0)) * Restrict UI server CORS to trusted origins ([#6986](#6986)) ([5e7ef3e](5e7ef3e)) * Return feature_view_metadata from the REST feature server ([#6927](#6927)) ([bc5aeef](bc5aeef)) * Serve aliased on demand feature views in feature services ([823b92e](823b92e)) * Snapshot tag filters in registry cache keys ([81e1546](81e1546)) * Stabilize materialization checks across hour boundaries ([#6948](#6948)) ([acadb17](acadb17)) * Stop the Couchbase online store reporting failed writes as written ([b3f3c98](b3f3c98)) * Support SQL registry dump ([b60f9c7](b60f9c7)) * Treat datetimes with a None utcoffset as naive ([687bee8](687bee8)) * Unwrap SecretStr tokens when building Trino auth objects ([#6771](#6771)) ([922db5c](922db5c)) * Use a valid ShuffleStrategy for the Ray DatasetContext ([6259f5c](6259f5c)) * Use do_exchange for offline server reads to support HPA ([bae07fc](bae07fc)) * Use group_by in Polars backend aggregation ([#6965](#6965)) ([7b9fad2](7b9fad2)) * Use upstream kubectl image as feast-operator CronJob default ([d6cc1dc](d6cc1dc)) * Use UTC end bound and correct docs for disable_event_timestamp ([#6956](#6956)) ([cc07c82](cc07c82)), closes [#6936](#6936) ### Features * Add a non-JVM read path for Lance data sources ([#6943](#6943)) ([e2a53bb](e2a53bb)), closes [#6925](#6925) [#5782](#5782) [#6909](#6909) * Add a non-JVM write path for Lance data sources ([#6945](#6945)) ([f754323](f754323)), closes [#6943](#6943) [#6909](#6909) * Add Chronon online and offline store integrations ([#6188](#6188)) ([dd059c6](dd059c6)) * Add consistency_level to the Milvus online store config ([97296c1](97296c1)) * Add dualStack server option to the Feast operator ([#6887](#6887)) ([725a147](725a147)), closes [#6977](#6977) * Add Lance optional dependencies ([#6958](#6958)) ([20bc993](20bc993)) * Add LanceFormat table format with version/tag pinning ([1196e22](1196e22)), closes [#5650](#5650) * Add MLflow as a first-class offline DataSource for Feast ([#6702](#6702)) ([3667772](3667772)) * Add optional GLIDE client for Redis online reads ([#6870](#6870)) ([6e9a532](6e9a532)) * Add quality monitoring features for Trino offline store ([#6779](#6779)) ([9340545](9340545)) * Add SavedDatasetLanceStorage ([1118d3e](1118d3e)) * Add standalone MCP server as a first-class component ([0efab5c](0efab5c)), closes [python-keycloak#623](https://github.com/python-keycloak/issues/623) * Add structured audit logging for MCP feature server ([#6456](#6456)) ([640ac70](640ac70)) * Add Trino compute engine for batch retrieval and materialization ([#6898](#6898)) ([11a47b4](11a47b4)) * Batch Postgres online reads across feature views ([#6832](#6832)) ([514bee7](514bee7)), closes [#3259](#3259) * **bigtable:** Make write batch size and concurrency configurable ([#6847](#6847)) ([5388522](5388522)) * Bucket feature count labels in latency metrics ([#6811](#6811)) ([d9ea120](d9ea120)) * Expose runtime version in Feast UI ([#6843](#6843)) ([fa8f06b](fa8f06b)) * Make Milvus index and search params configurable ([ef743ff](ef743ff)) * Offline write batch for IcebergSource ([5be5ed0](5be5ed0)) * OpenLineage consumer improvements - dataset versioning, column lineage, ownership ([69f294e](69f294e)) * **operator:** Add tolerations and nodeSelector to the FeatureStore CR ([#6741](#6741)) ([03dac6f](03dac6f)) * Serve the MCP server dual-stack on --host :: ([4fc9726](4fc9726)) * Support Milvus partition keys in the Milvus online store ([0ed3285](0ed3285)) * Support token, uri and db_name in the Milvus online store ([c5efecd](c5efecd)) * Version-pinning for FeatureService (online + offline) ([#6718](#6718)) ([787845a](787845a)) ### Performance Improvements * Avoid pandas copy in Spark reads ([#6878](#6878)) ([4771156](4771156)) * Decouple metrics HTTP server into a dedicated child process ([#6855](#6855)) ([0b621de](0b621de)) * Reduce online serving hot-path overhead from metrics and audit logging ([5ee7ee3](5ee7ee3))

* fix: Increase feast container probe timeouts to avoid false restarts Signed-off-by: Srihari <svenkata@redhat.com> * chore: Remove unused hermeto offline tree and sdist requirement locks (feast-dev#6987) * chore: Remove unused multicloud hermeto offline build tree Drop sdk/python/feast/infra/feature_servers/multicloud/offline/, which only supported local hermeto/arrow-from-source and RPM lock offline image builds and is unused by Makefile CI, GitHub Actions, or the standard feature-server Dockerfiles. Also clear related .secrets.baseline entries. Signed-off-by: ntkathole <nikhilkathole2683@gmail.com> Co-authored-by: Cursor <cursoragent@cursor.com> * chore: Drop unused minimal-sdist requirement locks Remove py3.10–3.12 minimal-sdist(-build) lockfiles, stop generating them in lock-python-dependencies-all, and delete the minimal-sdist-build extra. They only supported hermeto build-from-sdist offline image builds. Signed-off-by: ntkathole <nikhilkathole2683@gmail.com> Co-authored-by: Cursor <cursoragent@cursor.com> * chore: Refresh pixi.lock after removing minimal-sdist-build Keep pixi install --locked in sync with the pyproject.toml optional dependency change so ray/duckdb CI environments can install. Signed-off-by: ntkathole <nikhilkathole2683@gmail.com> Co-authored-by: Cursor <cursoragent@cursor.com> --------- Signed-off-by: ntkathole <nikhilkathole2683@gmail.com> Co-authored-by: Cursor <cursoragent@cursor.com> * chore(release): release 0.67.0 # [0.67.0](feast-dev/feast@v0.66.0...v0.67.0) (2026-10-13) ### Bug Fixes * Allow dill 0.4 so UDF serialization works on Python 3.14 ([feast-dev#6858](feast-dev#6858)) ([139704f](feast-dev@139704f)) * Allow serving features while a feature view is MATERIALIZING ([feast-dev#6789](feast-dev#6789)) ([e2d7fe2](feast-dev@e2d7fe2)), closes [feast-dev#6780](feast-dev#6780) * Allowlist the pinned commit SHA flagged by detect-secrets ([4b81d18](feast-dev@4b81d18)) * Apply point-in-time filter correctly with the Polars backend ([feast-dev#6992](feast-dev#6992)) ([01224d0](feast-dev@01224d0)) * Authorize registry server writes before deserializing UDFs ([e0ef16e](feast-dev@e0ef16e)) * Automate versioned documentation publication ([feast-dev#6763](feast-dev#6763)) ([5ad5592](feast-dev@5ad5592)), closes [feast-dev#6762](feast-dev#6762) * Avoid NumPy generic-unit deprecation when importing feast.type_map ([5fea698](feast-dev@5fea698)) * Bind metrics, REST registry, ui, and lineage servers dual-stack ([feast-dev#6886](feast-dev#6886)) ([8d2773c](feast-dev@8d2773c)) * Bound Arrow Flight batch size in the offline server ([feast-dev#6896](feast-dev#6896)) ([d169238](feast-dev@d169238)) * Compute std and var correctly for tiled aggregations ([213a130](feast-dev@213a130)) * Convert PdfBytes values to protos like ImageBytes ([feast-dev#6968](feast-dev#6968)) ([af20fe4](feast-dev@af20fe4)) * Decode fixed_size_list as the matching Feast list value type ([feast-dev#6944](feast-dev#6944)) ([8cfe891](feast-dev@8cfe891)) * Do not log /metrics requests from the pre-fork metrics server thread ([7087f1f](feast-dev@7087f1f)), closes [feast-dev#6928](feast-dev#6928) * Don't return from finally blocks in REST registry helpers ([d7cb4d0](feast-dev@d7cb4d0)) * Encode non-finite floats in feature server JSON responses ([feast-dev#6930](feast-dev#6930)) ([211ecb8](feast-dev@211ecb8)) * Feature freshness logic updated ([feast-dev#6765](feast-dev#6765)) ([f0bc070](feast-dev@f0bc070)) * Fix monitoring auth ([015cc84](feast-dev@015cc84)) * Handle non-nanosecond timestamps in tiled aggregations ([b8989cb](feast-dev@b8989cb)) * Improve feature list column sizing ([feast-dev#6793](feast-dev#6793)) ([2573693](feast-dev@2573693)) * Include vector attributes in Field equality ([feast-dev#6934](feast-dev#6934)) ([298c3f6](feast-dev@298c3f6)) * Increase feast container probe timeouts to avoid false restarts ([e8dbd50](feast-dev@e8dbd50)) * Infer the tensor type from retrieved values in to_tensor ([a03e323](feast-dev@a03e323)) * Infer timestamps for datetime columns in any unit or timezone ([810391f](feast-dev@810391f)) * Infer vector width when vector_length is undeclared ([0056565](feast-dev@0056565)), closes [feast-dev#6907](feast-dev#6907) * Initialize auth managers in standalone lineage server ([feast-dev#6873](feast-dev#6873)) ([ecd0d38](feast-dev@ecd0d38)) * Keep a zero watermark delay threshold on a Kafka source ([e2e28c7](feast-dev@e2e28c7)) * Keep entity rows apart in the ibis point-in-time join ([a87c070](feast-dev@a87c070)) * Keep the BatchFeatureView type and settings when copying it ([feast-dev#6989](feast-dev#6989)) ([013fc57](feast-dev@013fc57)) * Keep tiling, offline and stream engine settings when copying a StreamFeatureView ([fac0f88](feast-dev@fac0f88)) * Limit tiled max and min aggregations to the window ([2583c45](feast-dev@2583c45)) * Load Milvus collections once instead of on every query ([feast-dev#6882](feast-dev#6882)) ([f9e91fa](feast-dev@f9e91fa)) * Make vector length validation reachable, schema-driven and vectorized ([feast-dev#6909](feast-dev#6909)) ([29af8c0](feast-dev@29af8c0)), closes [feast-dev#6907](feast-dev#6907) * Map INT32_LIST, PDF_BYTES and IMAGE_BYTES type names to their ValueType ([8c1e825](feast-dev@8c1e825)) * Map narrow and unsigned Arrow integer types to Feast value types ([feast-dev#6960](feast-dev#6960)) ([ae74360](feast-dev@ae74360)) * Map Spark smallint and tinyint columns to INT32 ([9d42729](feast-dev@9d42729)) * Map SQL Server bigint to INT64 and float to DOUBLE ([6d15533](feast-dev@6d15533)) * Map Trino DECIMAL columns to pyarrow decimal128 ([155bc24](feast-dev@155bc24)) * Normalize Arrow-backed dtypes from Ray Dataset.to_pandas() for pyarrow >=25 compatibility ([feast-dev#6916](feast-dev#6916)) ([374f1d7](feast-dev@374f1d7)) * **operator:** Set grpc appProtocol on the offline Service port ([f9dcfb5](feast-dev@f9dcfb5)) * Pin expected tensor to same device in online retrieval test ([feast-dev#6777](feast-dev#6777)) ([1f2584d](feast-dev@1f2584d)) * Preserve an unset aggregation time window across a proto round-trip ([2000379](feast-dev@2000379)) * Preserve entity row order in precomputed reads ([ce37029](feast-dev@ce37029)) * Pull the MinIO test image from pgsty/minio ([fe27230](feast-dev@fe27230)) * Pull the MinIO test image from quay ([feast-dev#6838](feast-dev#6838)) ([367001b](feast-dev@367001b)) * Raise Trino monitoring query errors other than a missing table ([6161766](feast-dev@6161766)) * Read Kafka and Kinesis sources without a batch source from proto ([a620f5c](feast-dev@a620f5c)) * Read naive datetimes as UTC in UnixTimestamp values ([bc703d9](feast-dev@bc703d9)) * Record which storage class wrote a custom_storage payload ([feast-dev#6988](feast-dev#6988)) ([c0545f7](feast-dev@c0545f7)) * Refresh the SQL registry cache after deleting a data source ([66c55e3](feast-dev@66c55e3)) * Refresh the SQL registry cache after deleting a permission ([6b212f5](feast-dev@6b212f5)) * Reject unknown resource types and actions in the REST permission API ([a5eaa36](feast-dev@a5eaa36)) * Remove pip install command from Redis GLIDE docstring ([66ff990](feast-dev@66ff990)) * Remove runtime pip install hints from mlflow errors ([dd9f099](feast-dev@dd9f099)) * Remove unsupported vector_len from PostgreSQL tutorial config ([fcc9a85](feast-dev@fcc9a85)) * Repair Chronon operator validation and Pixi CI setup ([feast-dev#6947](feast-dev#6947)) ([3710f78](feast-dev@3710f78)) * Report Dask monitoring read errors instead of no data ([81673d9](feast-dev@81673d9)) * Resolve feature view batch source name against registry ([f8883d0](feast-dev@f8883d0)) * Restrict UI server CORS to trusted origins ([feast-dev#6986](feast-dev#6986)) ([5e7ef3e](feast-dev@5e7ef3e)) * Return feature_view_metadata from the REST feature server ([feast-dev#6927](feast-dev#6927)) ([bc5aeef](feast-dev@bc5aeef)) * Serve aliased on demand feature views in feature services ([823b92e](feast-dev@823b92e)) * Snapshot tag filters in registry cache keys ([81e1546](feast-dev@81e1546)) * Stabilize materialization checks across hour boundaries ([feast-dev#6948](feast-dev#6948)) ([acadb17](feast-dev@acadb17)) * Stop the Couchbase online store reporting failed writes as written ([b3f3c98](feast-dev@b3f3c98)) * Support SQL registry dump ([b60f9c7](feast-dev@b60f9c7)) * Treat datetimes with a None utcoffset as naive ([687bee8](feast-dev@687bee8)) * Unwrap SecretStr tokens when building Trino auth objects ([feast-dev#6771](feast-dev#6771)) ([922db5c](feast-dev@922db5c)) * Use a valid ShuffleStrategy for the Ray DatasetContext ([6259f5c](feast-dev@6259f5c)) * Use do_exchange for offline server reads to support HPA ([bae07fc](feast-dev@bae07fc)) * Use group_by in Polars backend aggregation ([feast-dev#6965](feast-dev#6965)) ([7b9fad2](feast-dev@7b9fad2)) * Use upstream kubectl image as feast-operator CronJob default ([d6cc1dc](feast-dev@d6cc1dc)) * Use UTC end bound and correct docs for disable_event_timestamp ([feast-dev#6956](feast-dev#6956)) ([cc07c82](feast-dev@cc07c82)), closes [feast-dev#6936](feast-dev#6936) ### Features * Add a non-JVM read path for Lance data sources ([feast-dev#6943](feast-dev#6943)) ([e2a53bb](feast-dev@e2a53bb)), closes [feast-dev#6925](feast-dev#6925) [feast-dev#5782](feast-dev#5782) [feast-dev#6909](feast-dev#6909) * Add a non-JVM write path for Lance data sources ([feast-dev#6945](feast-dev#6945)) ([f754323](feast-dev@f754323)), closes [feast-dev#6943](feast-dev#6943) [feast-dev#6909](feast-dev#6909) * Add Chronon online and offline store integrations ([feast-dev#6188](feast-dev#6188)) ([dd059c6](feast-dev@dd059c6)) * Add consistency_level to the Milvus online store config ([97296c1](feast-dev@97296c1)) * Add dualStack server option to the Feast operator ([feast-dev#6887](feast-dev#6887)) ([725a147](feast-dev@725a147)), closes [feast-dev#6977](feast-dev#6977) * Add Lance optional dependencies ([feast-dev#6958](feast-dev#6958)) ([20bc993](feast-dev@20bc993)) * Add LanceFormat table format with version/tag pinning ([1196e22](feast-dev@1196e22)), closes [feast-dev#5650](feast-dev#5650) * Add MLflow as a first-class offline DataSource for Feast ([feast-dev#6702](feast-dev#6702)) ([3667772](feast-dev@3667772)) * Add optional GLIDE client for Redis online reads ([feast-dev#6870](feast-dev#6870)) ([6e9a532](feast-dev@6e9a532)) * Add quality monitoring features for Trino offline store ([feast-dev#6779](feast-dev#6779)) ([9340545](feast-dev@9340545)) * Add SavedDatasetLanceStorage ([1118d3e](feast-dev@1118d3e)) * Add standalone MCP server as a first-class component ([0efab5c](feast-dev@0efab5c)), closes [python-keycloak#623](https://github.com/python-keycloak/issues/623) * Add structured audit logging for MCP feature server ([feast-dev#6456](feast-dev#6456)) ([640ac70](feast-dev@640ac70)) * Add Trino compute engine for batch retrieval and materialization ([feast-dev#6898](feast-dev#6898)) ([11a47b4](feast-dev@11a47b4)) * Batch Postgres online reads across feature views ([feast-dev#6832](feast-dev#6832)) ([514bee7](feast-dev@514bee7)), closes [feast-dev#3259](feast-dev#3259) * **bigtable:** Make write batch size and concurrency configurable ([feast-dev#6847](feast-dev#6847)) ([5388522](feast-dev@5388522)) * Bucket feature count labels in latency metrics ([feast-dev#6811](feast-dev#6811)) ([d9ea120](feast-dev@d9ea120)) * Expose runtime version in Feast UI ([feast-dev#6843](feast-dev#6843)) ([fa8f06b](feast-dev@fa8f06b)) * Make Milvus index and search params configurable ([ef743ff](feast-dev@ef743ff)) * Offline write batch for IcebergSource ([5be5ed0](feast-dev@5be5ed0)) * OpenLineage consumer improvements - dataset versioning, column lineage, ownership ([69f294e](feast-dev@69f294e)) * **operator:** Add tolerations and nodeSelector to the FeatureStore CR ([feast-dev#6741](feast-dev#6741)) ([03dac6f](feast-dev@03dac6f)) * Serve the MCP server dual-stack on --host :: ([4fc9726](feast-dev@4fc9726)) * Support Milvus partition keys in the Milvus online store ([0ed3285](feast-dev@0ed3285)) * Support token, uri and db_name in the Milvus online store ([c5efecd](feast-dev@c5efecd)) * Version-pinning for FeatureService (online + offline) ([feast-dev#6718](feast-dev#6718)) ([787845a](feast-dev@787845a)) ### Performance Improvements * Avoid pandas copy in Spark reads ([feast-dev#6878](feast-dev#6878)) ([4771156](feast-dev@4771156)) * Decouple metrics HTTP server into a dedicated child process ([feast-dev#6855](feast-dev#6855)) ([0b621de](feast-dev@0b621de)) * Reduce online serving hot-path overhead from metrics and audit logging ([5ee7ee3](feast-dev@5ee7ee3)) --------- Signed-off-by: Srihari <svenkata@redhat.com> Signed-off-by: ntkathole <nikhilkathole2683@gmail.com> Co-authored-by: Srihari <svenkata@redhat.com> Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: feast-ci-bot <feast-ci-bot@willem.co>

haoxu0 added 3 commits October 5, 2025 14:42

add support for table format such as Iceberg, Delta, Hudi etc.

8ae5ed5

Signed-off-by: HaoXuAI <sduxuhao@gmail.com>

linting

456b97f

Signed-off-by: HaoXuAI <sduxuhao@gmail.com>

linting

afa1444

Signed-off-by: HaoXuAI <sduxuhao@gmail.com>

haoxu0 requested a review from a team as a code owner October 5, 2025 21:47

haoxu0 changed the title ~~feat: Support table format~~ feat: Support table format: Iceberg, Delta, and Hudi Oct 5, 2025

haoxu0 added the ok-to-test label Oct 5, 2025

haoxu0 added 2 commits October 5, 2025 15:08

add tests

7a34eaf

Signed-off-by: HaoXuAI <sduxuhao@gmail.com>

fix tests

99f85b1

Signed-off-by: HaoXuAI <sduxuhao@gmail.com>

haoxu0 commented Oct 5, 2025

View reviewed changes

haoxu0 requested review from franciscojavierarceo and ntkathole October 5, 2025 22:34

franciscojavierarceo reviewed Oct 6, 2025

View reviewed changes

haoxu0 added 3 commits October 8, 2025 23:01

fix tests

eb15424

Signed-off-by: HaoXuAI <sduxuhao@gmail.com>

Merge branch 'master' into support-table-format

05e3a65

linting

b11c083

Signed-off-by: HaoXuAI <sduxuhao@gmail.com>

haoxu0 requested a review from franciscojavierarceo October 10, 2025 04:47

Merge branch 'master' into support-table-format

021f164

franciscojavierarceo requested changes Oct 10, 2025

View reviewed changes

haoxu0 closed this Oct 14, 2025

haoxu0 reopened this Oct 14, 2025

Merge branch 'master' into support-table-format

48b166b

haoxu0 and others added 7 commits October 29, 2025 21:36

Merge branch 'master' into support-table-format

429958a

add tableformat proto

e19467d

Signed-off-by: hao-xu5 <hxu44@apple.com>

update

931fdbd

Signed-off-by: hao-xu5 <hxu44@apple.com>

update doc

4919fba

Signed-off-by: hao-xu5 <hxu44@apple.com>

fix linting

63bba87

Signed-off-by: hao-xu5 <hxu44@apple.com>

Merge branch 'master' into support-table-format

b9bd58e

fix test

c014d6e

Signed-off-by: hao-xu5 <hxu44@apple.com>

franciscojavierarceo approved these changes Nov 6, 2025

View reviewed changes

haoxu0 merged commit 2915ad1 into master Nov 6, 2025
19 of 20 checks passed

franciscojavierarceo mentioned this pull request Jan 14, 2026

Add blog post on new transformation framework #5856

Open

Copilot AI mentioned this pull request Jan 14, 2026

docs: Add blog post on unified transformation framework #5858

Draft

haoxu0 mentioned this pull request Oct 1, 2026

feat: Add LanceFormat table format with version/tag pinning #6925

Merged

haoxu0 mentioned this pull request Oct 4, 2026

feat: Lance as an engine-agnostic, catalog-backed, version-pinned data source #6899

Open

Conversation

haoxu0 commented Oct 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What this PR does / why we need it:

Which issue(s) this PR fixes:

Misc

Uh oh!

haoxu0 Oct 5, 2025

Choose a reason for hiding this comment

Uh oh!

tokoko commented Oct 6, 2025

Uh oh!

franciscojavierarceo Oct 6, 2025

Choose a reason for hiding this comment

Uh oh!

franciscojavierarceo commented Oct 6, 2025

Uh oh!

haoxu0 commented Oct 6, 2025

Uh oh!

franciscojavierarceo commented Oct 7, 2025

Uh oh!

tokoko commented Oct 7, 2025

Uh oh!

haoxu0 commented Oct 9, 2025

Uh oh!

haoxu0 commented Oct 10, 2025

Uh oh!

franciscojavierarceo left a comment

Choose a reason for hiding this comment

Uh oh!

haoxu0 commented Oct 14, 2025

Uh oh!

haoxu0 commented Oct 14, 2025

Uh oh!

haoxu0 commented Nov 6, 2025

Uh oh!

franciscojavierarceo left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

haoxu0 commented Oct 5, 2025 •

edited

Loading