feat: Add LanceFormat table format with version/tag pinning by haoxu0 · Pull Request #6925 · feast-dev/feast

haoxu0 · 2026-10-01T20:27:19Z

Adds Lance to the existing TableFormat abstraction from #5650, rather than introducing a separate data source. Addresses the core of #6899.

Why `TableFormat` and not a new source

TableFormat already models Iceberg, Delta and Hudi as formats a source can carry, with catalog/namespace addressing and a properties bag. Lance fits that shape exactly, so it needs no new source class, no new offline store, and no changes to any existing consumer.

LanceFormat(
    catalog="my_catalog",
    namespace="my_namespace",
    version=3,            # or tag="candidate"
)

It works with `SparkSource` unchanged

SparkSource already drives its reader generically:

reader = spark_session.read.format(self.table_format.format_type.value)
for key, value in self.table_format.properties.items():
    reader = reader.option(key, value)

So mirroring the pin into properties is what makes this fall out for free. Verified:

SparkSource accepted it   : LanceFormat
spark .format(...) value  : lance
reader .option(...) pairs :
    lance.catalog = polaris_dev
    lance.namespace = ml_features
    lance.version = 3
SparkSource proto rt      : LanceFormat version = 3

Pin semantics

version / tag is the Lance-shaped instance of #5782. Two invariants:

version >= 1 is enforced. Lance dataset versions start at 1 and the proto treats 0 as unset, so without the guard version=0 would not round-trip — to_proto/from_proto read 0 as absent. Rejecting it keeps the Python and proto semantics in agreement rather than silently dropping a pin.
version and tag are mutually exclusive, since a tag already resolves to a version.

On the broader question @jfw-ppi raised in #5782 — whether a pin can change the response shape — the position I'd argue for is that a pin selects data, never shape: the declared FeatureView schema stays the contract, and a pinned version whose schema disagrees should fail explicitly. That belongs in whatever consumes the pin, so it is not in this PR, but the format carries enough information to enforce it.

Proto regeneration, deliberately constrained

Two things worth flagging, both about not doing the obvious thing.

1. Generated with grpcio-tools==1.62.3, not the pinned 1.84.0.

Regenerating with the pinned toolchain emits gencode that opens with:

_runtime_version.ValidateProtobufRuntimeVersion(_runtime_version.Domain.PUBLIC, 7, 35, 1, ...)

pyproject.toml declares protobuf>=4.24.0. That call would hard-fail for anyone on protobuf 4/5/6 — and google.protobuf.runtime_version does not exist in 4.x at all, so it is an ImportError on that one module while every other proto still imports. Using 1.62.3 emits 4.25.1-level gencode, matching every other checked-in proto, and keeps the declared floor honest.

Worth noting independently: the checked-in protos are at gencode 4.25.1 while the requirements pin grpcio-tools==1.84.0 / protobuf==7.36.2, so a full regeneration on master today would touch ~70 files and raise the effective protobuf floor. That looks like something to decide on purpose rather than as a side effect of a feature PR.

2. DataSource_pb2 is left untouched on purpose.

Regenerating also rewrites DataSource_pb2.py/.pyi (~125 lines), but that churn is pre-existing — I verified it reproduces on pristine master with no changes at all. The checked-in copy is missing _CONNECTIONREF_PARAMSENTRY entries, i.e. it is stale against DataSource.proto. Happy to fix that separately; it does not belong here.

Net result is 5 files, and only DataFormat_pb2 regenerated.

Scope

This adds the format descriptor. It does not add a Lance reader or offline store — that is the follow-on discussed in #6899, and it is why the tests here cover LanceFormat semantics rather than reading real datasets (no new test dependency on pylance).

Also explicitly not proposing Lance as an online store: it is a format plus indexes, not a low-latency KV service, and its write path is columnar while online_write_batch is row-oriented proto.

Testing

11 new tests in test_table_format.py (24 total in the file): creation, minimal construction, version pin, tag pin, version < 1 rejection, version+tag rejection, dict/json/proto round-trips, unpinned not coming back as version 0, and factory dispatch.

test_table_format.py — 24 passed
test_utils.py, test_data_sources.py, test_types.py — 52 passed, 1 skipped
mypy feast/table_format.py — clean, no issues
ruff check / ruff format --check — clean

Related: #6899, #5782, #5650, #6499

codecov-commenter · 2026-10-01T20:30:18Z

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

❌ Patch coverage is 97.61905% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 48.68%. Comparing base (298c3f6) to head (9afb378).
⚠️ Report is 3 commits behind head on master.

Files with missing lines	Patch %	Lines
sdk/python/feast/table_format.py	97.61%	0 Missing and 1 partial ⚠️
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #6925      +/-   ##
==========================================
+ Coverage   48.64%   48.68%   +0.04%     
==========================================
  Files         427      427              
  Lines       53864    53906      +42     
  Branches     7849     7858       +9     
==========================================
+ Hits        26203    26246      +43     
+ Misses      25792    25788       -4     
- Partials     1869     1872       +3

Flag	Coverage Δ
go-feature-server	`30.58% <ø> (ø)`
python-unit	`50.06% <97.61%> (+0.04%)`	⬆️

Files with missing lines	Coverage Δ
sdk/python/feast/table_format.py	`85.62% <97.61%> (+5.11%)`	⬆️

... and 1 file with indirect coverage changes

Continue to review full report in Codecov by Harness.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 298c3f6...9afb378. Read the comment docs.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

haoxu0 · 2026-10-02T20:59:24Z

@ntkathole PTAL, It's a simple one

Extends the existing TableFormat abstraction (feast-dev#5650) with Lance rather than introducing a separate data source, so Lance is addressed the same way Iceberg, Delta and Hudi already are. Closes part of feast-dev#6899. LanceFormat carries catalog/namespace addressing plus an optional pin to a dataset version or tag. Because SparkSource already drives its reader generically from table_format.format_type.value and table_format.properties, this works with SparkSource with no changes to it: format_type.value is "lance", and the pin is mirrored into properties as lance.version / lance.tag. version is validated as >= 1 so the Python and proto semantics agree. Lance dataset versions start at 1 and the proto treats 0 as unset, so without that guard version=0 would not round-trip, since to_proto/from_proto read 0 as absent. version and tag are mutually exclusive, because a tag already resolves to a version. Only DataFormat_pb2 is regenerated, using grpcio-tools 1.62.3 so the emitted gencode stays at the 4.25.1 level the other checked-in protos use. Regenerating with the pinned grpcio-tools 1.84.0 instead emits gencode that calls ValidateProtobufRuntimeVersion for protobuf 7.35.1, which would break the declared protobuf>=4.24.0 floor for that one module. DataSource_pb2 is deliberately left untouched. It is already stale against DataSource.proto on master, missing ConnectionRef entries, and regenerating it produces ~125 lines of churn unrelated to this change. Signed-off-by: hao-xu5 <hxu44@apple.com>

Adds LanceSource and teaches the DuckDB offline store to read it, completing the read half of feast-dev#6899. LanceFormat landed in feast-dev#6925 as a format descriptor; nothing read Lance until now. Lance already worked through SparkSource, which drives its reader generically from table_format.format_type.value and table_format.properties. What was missing is a path that needs no JVM, which is also the real test of whether the DataSource abstraction is engine-agnostic rather than Spark-agnostic in name only. Placement: a new source read by the existing DuckDB store, rather than a Lance offline store or an extension of FileSource. FileSource is the wrong host. Its format axis is already taken by file_format, so adding table_format would give one source two overlapping format axes. It is also read by two stores with incompatible contracts: duckdb._read_data_source dispatches on type, while dask._read_datasource has no dispatch seam and reads file_options.uri unconditionally as Parquet, and asserts isinstance(..., FileSource) in three places. Decisively, Lance's catalog addressing has no path to put in FileSource.path, so the catalog-based layer would not fit the class even if the path-based one did. A Lance offline store would be the wrong 120 lines. duckdb.py is a binding that injects reader and writer callbacks into the engine in ibis.py, so a Lance store would be a near-copy of it plus a repo_config entry, and would force a choice between Lance and Parquet instead of mixing them in one feature service. Reading a source as ibis.memtable(arrow_table) in the DuckDB store already has two precedents, IcebergSource and MlflowDatasetSource. Following them leaves ibis.py untouched, so the point-in-time join, TTL handling, field mapping and ODFVs work unchanged, and no edit to repo_config.py or data_source.py is needed because CUSTOM_SOURCE plus data_source_class_type is self-describing. Both addressing modes work: a uri, and catalog/namespace/table through namespace_client and table_id. Pin semantics follow what was argued on feast-dev#5782 and feast-dev#6925: a pin selects data, never shape. get_table_column_names_and_types reads the pinned schema so feast apply infers what reads will actually see; a pre-flight check fails with a message naming the pin when a pinned version cannot satisfy the declared schema; and vector widths go through _validate_vector_field_lengths from feast-dev#6909 rather than a second validator. Tests use the dir namespace implementation, which exercises the same namespace_client and table_id code path as a remote catalog with no server required. 43 tests, including a demonstration that a tag pin returns earlier data after the dataset has been overwritten for the same entity and timestamp. Read-only for now: _write_data_source is untouched, so a LanceSource is not yet a persist target and there is no SavedDatasetLanceStorage. Signed-off-by: hao-xu5 <hxu44@apple.com>

* feat: Add a non-JVM read path for Lance data sources Adds LanceSource and teaches the DuckDB offline store to read it, completing the read half of #6899. LanceFormat landed in #6925 as a format descriptor; nothing read Lance until now. Lance already worked through SparkSource, which drives its reader generically from table_format.format_type.value and table_format.properties. What was missing is a path that needs no JVM, which is also the real test of whether the DataSource abstraction is engine-agnostic rather than Spark-agnostic in name only. Placement: a new source read by the existing DuckDB store, rather than a Lance offline store or an extension of FileSource. FileSource is the wrong host. Its format axis is already taken by file_format, so adding table_format would give one source two overlapping format axes. It is also read by two stores with incompatible contracts: duckdb._read_data_source dispatches on type, while dask._read_datasource has no dispatch seam and reads file_options.uri unconditionally as Parquet, and asserts isinstance(..., FileSource) in three places. Decisively, Lance's catalog addressing has no path to put in FileSource.path, so the catalog-based layer would not fit the class even if the path-based one did. A Lance offline store would be the wrong 120 lines. duckdb.py is a binding that injects reader and writer callbacks into the engine in ibis.py, so a Lance store would be a near-copy of it plus a repo_config entry, and would force a choice between Lance and Parquet instead of mixing them in one feature service. Reading a source as ibis.memtable(arrow_table) in the DuckDB store already has two precedents, IcebergSource and MlflowDatasetSource. Following them leaves ibis.py untouched, so the point-in-time join, TTL handling, field mapping and ODFVs work unchanged, and no edit to repo_config.py or data_source.py is needed because CUSTOM_SOURCE plus data_source_class_type is self-describing. Both addressing modes work: a uri, and catalog/namespace/table through namespace_client and table_id. Pin semantics follow what was argued on #5782 and #6925: a pin selects data, never shape. get_table_column_names_and_types reads the pinned schema so feast apply infers what reads will actually see; a pre-flight check fails with a message naming the pin when a pinned version cannot satisfy the declared schema; and vector widths go through _validate_vector_field_lengths from #6909 rather than a second validator. Tests use the dir namespace implementation, which exercises the same namespace_client and table_id code path as a remote catalog with no server required. 43 tests, including a demonstration that a tag pin returns earlier data after the dataset has been overwritten for the same entity and timestamp. Read-only for now: _write_data_source is untouched, so a LanceSource is not yet a persist target and there is no SavedDatasetLanceStorage. Signed-off-by: hao-xu5 <hxu44@apple.com> * fix: Address Lance read review feedback Signed-off-by: HaoXuAI <sduxuhao@gmail.com> --------- Signed-off-by: hao-xu5 <hxu44@apple.com> Signed-off-by: HaoXuAI <sduxuhao@gmail.com>

# [0.67.0](v0.66.0...v0.67.0) (2026-10-13) ### Bug Fixes * Allow dill 0.4 so UDF serialization works on Python 3.14 ([#6858](#6858)) ([139704f](139704f)) * Allow serving features while a feature view is MATERIALIZING ([#6789](#6789)) ([e2d7fe2](e2d7fe2)), closes [#6780](#6780) * Allowlist the pinned commit SHA flagged by detect-secrets ([4b81d18](4b81d18)) * Apply point-in-time filter correctly with the Polars backend ([#6992](#6992)) ([01224d0](01224d0)) * Authorize registry server writes before deserializing UDFs ([e0ef16e](e0ef16e)) * Automate versioned documentation publication ([#6763](#6763)) ([5ad5592](5ad5592)), closes [#6762](#6762) * Avoid NumPy generic-unit deprecation when importing feast.type_map ([5fea698](5fea698)) * Bind metrics, REST registry, ui, and lineage servers dual-stack ([#6886](#6886)) ([8d2773c](8d2773c)) * Bound Arrow Flight batch size in the offline server ([#6896](#6896)) ([d169238](d169238)) * Compute std and var correctly for tiled aggregations ([213a130](213a130)) * Convert PdfBytes values to protos like ImageBytes ([#6968](#6968)) ([af20fe4](af20fe4)) * Decode fixed_size_list as the matching Feast list value type ([#6944](#6944)) ([8cfe891](8cfe891)) * Do not log /metrics requests from the pre-fork metrics server thread ([7087f1f](7087f1f)), closes [#6928](#6928) * Don't return from finally blocks in REST registry helpers ([d7cb4d0](d7cb4d0)) * Encode non-finite floats in feature server JSON responses ([#6930](#6930)) ([211ecb8](211ecb8)) * Feature freshness logic updated ([#6765](#6765)) ([f0bc070](f0bc070)) * Fix monitoring auth ([015cc84](015cc84)) * Handle non-nanosecond timestamps in tiled aggregations ([b8989cb](b8989cb)) * Improve feature list column sizing ([#6793](#6793)) ([2573693](2573693)) * Include vector attributes in Field equality ([#6934](#6934)) ([298c3f6](298c3f6)) * Increase feast container probe timeouts to avoid false restarts ([e8dbd50](e8dbd50)) * Infer the tensor type from retrieved values in to_tensor ([a03e323](a03e323)) * Infer timestamps for datetime columns in any unit or timezone ([810391f](810391f)) * Infer vector width when vector_length is undeclared ([0056565](0056565)), closes [#6907](#6907) * Initialize auth managers in standalone lineage server ([#6873](#6873)) ([ecd0d38](ecd0d38)) * Keep a zero watermark delay threshold on a Kafka source ([e2e28c7](e2e28c7)) * Keep entity rows apart in the ibis point-in-time join ([a87c070](a87c070)) * Keep the BatchFeatureView type and settings when copying it ([#6989](#6989)) ([013fc57](013fc57)) * Keep tiling, offline and stream engine settings when copying a StreamFeatureView ([fac0f88](fac0f88)) * Limit tiled max and min aggregations to the window ([2583c45](2583c45)) * Load Milvus collections once instead of on every query ([#6882](#6882)) ([f9e91fa](f9e91fa)) * Make vector length validation reachable, schema-driven and vectorized ([#6909](#6909)) ([29af8c0](29af8c0)), closes [#6907](#6907) * Map INT32_LIST, PDF_BYTES and IMAGE_BYTES type names to their ValueType ([8c1e825](8c1e825)) * Map narrow and unsigned Arrow integer types to Feast value types ([#6960](#6960)) ([ae74360](ae74360)) * Map Spark smallint and tinyint columns to INT32 ([9d42729](9d42729)) * Map SQL Server bigint to INT64 and float to DOUBLE ([6d15533](6d15533)) * Map Trino DECIMAL columns to pyarrow decimal128 ([155bc24](155bc24)) * Normalize Arrow-backed dtypes from Ray Dataset.to_pandas() for pyarrow >=25 compatibility ([#6916](#6916)) ([374f1d7](374f1d7)) * **operator:** Set grpc appProtocol on the offline Service port ([f9dcfb5](f9dcfb5)) * Pin expected tensor to same device in online retrieval test ([#6777](#6777)) ([1f2584d](1f2584d)) * Preserve an unset aggregation time window across a proto round-trip ([2000379](2000379)) * Preserve entity row order in precomputed reads ([ce37029](ce37029)) * Pull the MinIO test image from pgsty/minio ([fe27230](fe27230)) * Pull the MinIO test image from quay ([#6838](#6838)) ([367001b](367001b)) * Raise Trino monitoring query errors other than a missing table ([6161766](6161766)) * Read Kafka and Kinesis sources without a batch source from proto ([a620f5c](a620f5c)) * Read naive datetimes as UTC in UnixTimestamp values ([bc703d9](bc703d9)) * Record which storage class wrote a custom_storage payload ([#6988](#6988)) ([c0545f7](c0545f7)) * Refresh the SQL registry cache after deleting a data source ([66c55e3](66c55e3)) * Refresh the SQL registry cache after deleting a permission ([6b212f5](6b212f5)) * Reject unknown resource types and actions in the REST permission API ([a5eaa36](a5eaa36)) * Remove pip install command from Redis GLIDE docstring ([66ff990](66ff990)) * Remove runtime pip install hints from mlflow errors ([dd9f099](dd9f099)) * Remove unsupported vector_len from PostgreSQL tutorial config ([fcc9a85](fcc9a85)) * Repair Chronon operator validation and Pixi CI setup ([#6947](#6947)) ([3710f78](3710f78)) * Report Dask monitoring read errors instead of no data ([81673d9](81673d9)) * Resolve feature view batch source name against registry ([f8883d0](f8883d0)) * Restrict UI server CORS to trusted origins ([#6986](#6986)) ([5e7ef3e](5e7ef3e)) * Return feature_view_metadata from the REST feature server ([#6927](#6927)) ([bc5aeef](bc5aeef)) * Serve aliased on demand feature views in feature services ([823b92e](823b92e)) * Snapshot tag filters in registry cache keys ([81e1546](81e1546)) * Stabilize materialization checks across hour boundaries ([#6948](#6948)) ([acadb17](acadb17)) * Stop the Couchbase online store reporting failed writes as written ([b3f3c98](b3f3c98)) * Support SQL registry dump ([b60f9c7](b60f9c7)) * Treat datetimes with a None utcoffset as naive ([687bee8](687bee8)) * Unwrap SecretStr tokens when building Trino auth objects ([#6771](#6771)) ([922db5c](922db5c)) * Use a valid ShuffleStrategy for the Ray DatasetContext ([6259f5c](6259f5c)) * Use do_exchange for offline server reads to support HPA ([bae07fc](bae07fc)) * Use group_by in Polars backend aggregation ([#6965](#6965)) ([7b9fad2](7b9fad2)) * Use upstream kubectl image as feast-operator CronJob default ([d6cc1dc](d6cc1dc)) * Use UTC end bound and correct docs for disable_event_timestamp ([#6956](#6956)) ([cc07c82](cc07c82)), closes [#6936](#6936) ### Features * Add a non-JVM read path for Lance data sources ([#6943](#6943)) ([e2a53bb](e2a53bb)), closes [#6925](#6925) [#5782](#5782) [#6909](#6909) * Add a non-JVM write path for Lance data sources ([#6945](#6945)) ([f754323](f754323)), closes [#6943](#6943) [#6909](#6909) * Add Chronon online and offline store integrations ([#6188](#6188)) ([dd059c6](dd059c6)) * Add consistency_level to the Milvus online store config ([97296c1](97296c1)) * Add dualStack server option to the Feast operator ([#6887](#6887)) ([725a147](725a147)), closes [#6977](#6977) * Add Lance optional dependencies ([#6958](#6958)) ([20bc993](20bc993)) * Add LanceFormat table format with version/tag pinning ([1196e22](1196e22)), closes [#5650](#5650) * Add MLflow as a first-class offline DataSource for Feast ([#6702](#6702)) ([3667772](3667772)) * Add optional GLIDE client for Redis online reads ([#6870](#6870)) ([6e9a532](6e9a532)) * Add quality monitoring features for Trino offline store ([#6779](#6779)) ([9340545](9340545)) * Add SavedDatasetLanceStorage ([1118d3e](1118d3e)) * Add standalone MCP server as a first-class component ([0efab5c](0efab5c)), closes [python-keycloak#623](https://github.com/python-keycloak/issues/623) * Add structured audit logging for MCP feature server ([#6456](#6456)) ([640ac70](640ac70)) * Add Trino compute engine for batch retrieval and materialization ([#6898](#6898)) ([11a47b4](11a47b4)) * Batch Postgres online reads across feature views ([#6832](#6832)) ([514bee7](514bee7)), closes [#3259](#3259) * **bigtable:** Make write batch size and concurrency configurable ([#6847](#6847)) ([5388522](5388522)) * Bucket feature count labels in latency metrics ([#6811](#6811)) ([d9ea120](d9ea120)) * Expose runtime version in Feast UI ([#6843](#6843)) ([fa8f06b](fa8f06b)) * Make Milvus index and search params configurable ([ef743ff](ef743ff)) * Offline write batch for IcebergSource ([5be5ed0](5be5ed0)) * OpenLineage consumer improvements - dataset versioning, column lineage, ownership ([69f294e](69f294e)) * **operator:** Add tolerations and nodeSelector to the FeatureStore CR ([#6741](#6741)) ([03dac6f](03dac6f)) * Serve the MCP server dual-stack on --host :: ([4fc9726](4fc9726)) * Support Milvus partition keys in the Milvus online store ([0ed3285](0ed3285)) * Support token, uri and db_name in the Milvus online store ([c5efecd](c5efecd)) * Version-pinning for FeatureService (online + offline) ([#6718](#6718)) ([787845a](787845a)) ### Performance Improvements * Avoid pandas copy in Spark reads ([#6878](#6878)) ([4771156](4771156)) * Decouple metrics HTTP server into a dedicated child process ([#6855](#6855)) ([0b621de](0b621de)) * Reduce online serving hot-path overhead from metrics and audit logging ([5ee7ee3](5ee7ee3))

* fix: Increase feast container probe timeouts to avoid false restarts Signed-off-by: Srihari <svenkata@redhat.com> * chore: Remove unused hermeto offline tree and sdist requirement locks (feast-dev#6987) * chore: Remove unused multicloud hermeto offline build tree Drop sdk/python/feast/infra/feature_servers/multicloud/offline/, which only supported local hermeto/arrow-from-source and RPM lock offline image builds and is unused by Makefile CI, GitHub Actions, or the standard feature-server Dockerfiles. Also clear related .secrets.baseline entries. Signed-off-by: ntkathole <nikhilkathole2683@gmail.com> Co-authored-by: Cursor <cursoragent@cursor.com> * chore: Drop unused minimal-sdist requirement locks Remove py3.10–3.12 minimal-sdist(-build) lockfiles, stop generating them in lock-python-dependencies-all, and delete the minimal-sdist-build extra. They only supported hermeto build-from-sdist offline image builds. Signed-off-by: ntkathole <nikhilkathole2683@gmail.com> Co-authored-by: Cursor <cursoragent@cursor.com> * chore: Refresh pixi.lock after removing minimal-sdist-build Keep pixi install --locked in sync with the pyproject.toml optional dependency change so ray/duckdb CI environments can install. Signed-off-by: ntkathole <nikhilkathole2683@gmail.com> Co-authored-by: Cursor <cursoragent@cursor.com> --------- Signed-off-by: ntkathole <nikhilkathole2683@gmail.com> Co-authored-by: Cursor <cursoragent@cursor.com> * chore(release): release 0.67.0 # [0.67.0](feast-dev/feast@v0.66.0...v0.67.0) (2026-10-13) ### Bug Fixes * Allow dill 0.4 so UDF serialization works on Python 3.14 ([feast-dev#6858](feast-dev#6858)) ([139704f](feast-dev@139704f)) * Allow serving features while a feature view is MATERIALIZING ([feast-dev#6789](feast-dev#6789)) ([e2d7fe2](feast-dev@e2d7fe2)), closes [feast-dev#6780](feast-dev#6780) * Allowlist the pinned commit SHA flagged by detect-secrets ([4b81d18](feast-dev@4b81d18)) * Apply point-in-time filter correctly with the Polars backend ([feast-dev#6992](feast-dev#6992)) ([01224d0](feast-dev@01224d0)) * Authorize registry server writes before deserializing UDFs ([e0ef16e](feast-dev@e0ef16e)) * Automate versioned documentation publication ([feast-dev#6763](feast-dev#6763)) ([5ad5592](feast-dev@5ad5592)), closes [feast-dev#6762](feast-dev#6762) * Avoid NumPy generic-unit deprecation when importing feast.type_map ([5fea698](feast-dev@5fea698)) * Bind metrics, REST registry, ui, and lineage servers dual-stack ([feast-dev#6886](feast-dev#6886)) ([8d2773c](feast-dev@8d2773c)) * Bound Arrow Flight batch size in the offline server ([feast-dev#6896](feast-dev#6896)) ([d169238](feast-dev@d169238)) * Compute std and var correctly for tiled aggregations ([213a130](feast-dev@213a130)) * Convert PdfBytes values to protos like ImageBytes ([feast-dev#6968](feast-dev#6968)) ([af20fe4](feast-dev@af20fe4)) * Decode fixed_size_list as the matching Feast list value type ([feast-dev#6944](feast-dev#6944)) ([8cfe891](feast-dev@8cfe891)) * Do not log /metrics requests from the pre-fork metrics server thread ([7087f1f](feast-dev@7087f1f)), closes [feast-dev#6928](feast-dev#6928) * Don't return from finally blocks in REST registry helpers ([d7cb4d0](feast-dev@d7cb4d0)) * Encode non-finite floats in feature server JSON responses ([feast-dev#6930](feast-dev#6930)) ([211ecb8](feast-dev@211ecb8)) * Feature freshness logic updated ([feast-dev#6765](feast-dev#6765)) ([f0bc070](feast-dev@f0bc070)) * Fix monitoring auth ([015cc84](feast-dev@015cc84)) * Handle non-nanosecond timestamps in tiled aggregations ([b8989cb](feast-dev@b8989cb)) * Improve feature list column sizing ([feast-dev#6793](feast-dev#6793)) ([2573693](feast-dev@2573693)) * Include vector attributes in Field equality ([feast-dev#6934](feast-dev#6934)) ([298c3f6](feast-dev@298c3f6)) * Increase feast container probe timeouts to avoid false restarts ([e8dbd50](feast-dev@e8dbd50)) * Infer the tensor type from retrieved values in to_tensor ([a03e323](feast-dev@a03e323)) * Infer timestamps for datetime columns in any unit or timezone ([810391f](feast-dev@810391f)) * Infer vector width when vector_length is undeclared ([0056565](feast-dev@0056565)), closes [feast-dev#6907](feast-dev#6907) * Initialize auth managers in standalone lineage server ([feast-dev#6873](feast-dev#6873)) ([ecd0d38](feast-dev@ecd0d38)) * Keep a zero watermark delay threshold on a Kafka source ([e2e28c7](feast-dev@e2e28c7)) * Keep entity rows apart in the ibis point-in-time join ([a87c070](feast-dev@a87c070)) * Keep the BatchFeatureView type and settings when copying it ([feast-dev#6989](feast-dev#6989)) ([013fc57](feast-dev@013fc57)) * Keep tiling, offline and stream engine settings when copying a StreamFeatureView ([fac0f88](feast-dev@fac0f88)) * Limit tiled max and min aggregations to the window ([2583c45](feast-dev@2583c45)) * Load Milvus collections once instead of on every query ([feast-dev#6882](feast-dev#6882)) ([f9e91fa](feast-dev@f9e91fa)) * Make vector length validation reachable, schema-driven and vectorized ([feast-dev#6909](feast-dev#6909)) ([29af8c0](feast-dev@29af8c0)), closes [feast-dev#6907](feast-dev#6907) * Map INT32_LIST, PDF_BYTES and IMAGE_BYTES type names to their ValueType ([8c1e825](feast-dev@8c1e825)) * Map narrow and unsigned Arrow integer types to Feast value types ([feast-dev#6960](feast-dev#6960)) ([ae74360](feast-dev@ae74360)) * Map Spark smallint and tinyint columns to INT32 ([9d42729](feast-dev@9d42729)) * Map SQL Server bigint to INT64 and float to DOUBLE ([6d15533](feast-dev@6d15533)) * Map Trino DECIMAL columns to pyarrow decimal128 ([155bc24](feast-dev@155bc24)) * Normalize Arrow-backed dtypes from Ray Dataset.to_pandas() for pyarrow >=25 compatibility ([feast-dev#6916](feast-dev#6916)) ([374f1d7](feast-dev@374f1d7)) * **operator:** Set grpc appProtocol on the offline Service port ([f9dcfb5](feast-dev@f9dcfb5)) * Pin expected tensor to same device in online retrieval test ([feast-dev#6777](feast-dev#6777)) ([1f2584d](feast-dev@1f2584d)) * Preserve an unset aggregation time window across a proto round-trip ([2000379](feast-dev@2000379)) * Preserve entity row order in precomputed reads ([ce37029](feast-dev@ce37029)) * Pull the MinIO test image from pgsty/minio ([fe27230](feast-dev@fe27230)) * Pull the MinIO test image from quay ([feast-dev#6838](feast-dev#6838)) ([367001b](feast-dev@367001b)) * Raise Trino monitoring query errors other than a missing table ([6161766](feast-dev@6161766)) * Read Kafka and Kinesis sources without a batch source from proto ([a620f5c](feast-dev@a620f5c)) * Read naive datetimes as UTC in UnixTimestamp values ([bc703d9](feast-dev@bc703d9)) * Record which storage class wrote a custom_storage payload ([feast-dev#6988](feast-dev#6988)) ([c0545f7](feast-dev@c0545f7)) * Refresh the SQL registry cache after deleting a data source ([66c55e3](feast-dev@66c55e3)) * Refresh the SQL registry cache after deleting a permission ([6b212f5](feast-dev@6b212f5)) * Reject unknown resource types and actions in the REST permission API ([a5eaa36](feast-dev@a5eaa36)) * Remove pip install command from Redis GLIDE docstring ([66ff990](feast-dev@66ff990)) * Remove runtime pip install hints from mlflow errors ([dd9f099](feast-dev@dd9f099)) * Remove unsupported vector_len from PostgreSQL tutorial config ([fcc9a85](feast-dev@fcc9a85)) * Repair Chronon operator validation and Pixi CI setup ([feast-dev#6947](feast-dev#6947)) ([3710f78](feast-dev@3710f78)) * Report Dask monitoring read errors instead of no data ([81673d9](feast-dev@81673d9)) * Resolve feature view batch source name against registry ([f8883d0](feast-dev@f8883d0)) * Restrict UI server CORS to trusted origins ([feast-dev#6986](feast-dev#6986)) ([5e7ef3e](feast-dev@5e7ef3e)) * Return feature_view_metadata from the REST feature server ([feast-dev#6927](feast-dev#6927)) ([bc5aeef](feast-dev@bc5aeef)) * Serve aliased on demand feature views in feature services ([823b92e](feast-dev@823b92e)) * Snapshot tag filters in registry cache keys ([81e1546](feast-dev@81e1546)) * Stabilize materialization checks across hour boundaries ([feast-dev#6948](feast-dev#6948)) ([acadb17](feast-dev@acadb17)) * Stop the Couchbase online store reporting failed writes as written ([b3f3c98](feast-dev@b3f3c98)) * Support SQL registry dump ([b60f9c7](feast-dev@b60f9c7)) * Treat datetimes with a None utcoffset as naive ([687bee8](feast-dev@687bee8)) * Unwrap SecretStr tokens when building Trino auth objects ([feast-dev#6771](feast-dev#6771)) ([922db5c](feast-dev@922db5c)) * Use a valid ShuffleStrategy for the Ray DatasetContext ([6259f5c](feast-dev@6259f5c)) * Use do_exchange for offline server reads to support HPA ([bae07fc](feast-dev@bae07fc)) * Use group_by in Polars backend aggregation ([feast-dev#6965](feast-dev#6965)) ([7b9fad2](feast-dev@7b9fad2)) * Use upstream kubectl image as feast-operator CronJob default ([d6cc1dc](feast-dev@d6cc1dc)) * Use UTC end bound and correct docs for disable_event_timestamp ([feast-dev#6956](feast-dev#6956)) ([cc07c82](feast-dev@cc07c82)), closes [feast-dev#6936](feast-dev#6936) ### Features * Add a non-JVM read path for Lance data sources ([feast-dev#6943](feast-dev#6943)) ([e2a53bb](feast-dev@e2a53bb)), closes [feast-dev#6925](feast-dev#6925) [feast-dev#5782](feast-dev#5782) [feast-dev#6909](feast-dev#6909) * Add a non-JVM write path for Lance data sources ([feast-dev#6945](feast-dev#6945)) ([f754323](feast-dev@f754323)), closes [feast-dev#6943](feast-dev#6943) [feast-dev#6909](feast-dev#6909) * Add Chronon online and offline store integrations ([feast-dev#6188](feast-dev#6188)) ([dd059c6](feast-dev@dd059c6)) * Add consistency_level to the Milvus online store config ([97296c1](feast-dev@97296c1)) * Add dualStack server option to the Feast operator ([feast-dev#6887](feast-dev#6887)) ([725a147](feast-dev@725a147)), closes [feast-dev#6977](feast-dev#6977) * Add Lance optional dependencies ([feast-dev#6958](feast-dev#6958)) ([20bc993](feast-dev@20bc993)) * Add LanceFormat table format with version/tag pinning ([1196e22](feast-dev@1196e22)), closes [feast-dev#5650](feast-dev#5650) * Add MLflow as a first-class offline DataSource for Feast ([feast-dev#6702](feast-dev#6702)) ([3667772](feast-dev@3667772)) * Add optional GLIDE client for Redis online reads ([feast-dev#6870](feast-dev#6870)) ([6e9a532](feast-dev@6e9a532)) * Add quality monitoring features for Trino offline store ([feast-dev#6779](feast-dev#6779)) ([9340545](feast-dev@9340545)) * Add SavedDatasetLanceStorage ([1118d3e](feast-dev@1118d3e)) * Add standalone MCP server as a first-class component ([0efab5c](feast-dev@0efab5c)), closes [python-keycloak#623](https://github.com/python-keycloak/issues/623) * Add structured audit logging for MCP feature server ([feast-dev#6456](feast-dev#6456)) ([640ac70](feast-dev@640ac70)) * Add Trino compute engine for batch retrieval and materialization ([feast-dev#6898](feast-dev#6898)) ([11a47b4](feast-dev@11a47b4)) * Batch Postgres online reads across feature views ([feast-dev#6832](feast-dev#6832)) ([514bee7](feast-dev@514bee7)), closes [feast-dev#3259](feast-dev#3259) * **bigtable:** Make write batch size and concurrency configurable ([feast-dev#6847](feast-dev#6847)) ([5388522](feast-dev@5388522)) * Bucket feature count labels in latency metrics ([feast-dev#6811](feast-dev#6811)) ([d9ea120](feast-dev@d9ea120)) * Expose runtime version in Feast UI ([feast-dev#6843](feast-dev#6843)) ([fa8f06b](feast-dev@fa8f06b)) * Make Milvus index and search params configurable ([ef743ff](feast-dev@ef743ff)) * Offline write batch for IcebergSource ([5be5ed0](feast-dev@5be5ed0)) * OpenLineage consumer improvements - dataset versioning, column lineage, ownership ([69f294e](feast-dev@69f294e)) * **operator:** Add tolerations and nodeSelector to the FeatureStore CR ([feast-dev#6741](feast-dev#6741)) ([03dac6f](feast-dev@03dac6f)) * Serve the MCP server dual-stack on --host :: ([4fc9726](feast-dev@4fc9726)) * Support Milvus partition keys in the Milvus online store ([0ed3285](feast-dev@0ed3285)) * Support token, uri and db_name in the Milvus online store ([c5efecd](feast-dev@c5efecd)) * Version-pinning for FeatureService (online + offline) ([feast-dev#6718](feast-dev#6718)) ([787845a](feast-dev@787845a)) ### Performance Improvements * Avoid pandas copy in Spark reads ([feast-dev#6878](feast-dev#6878)) ([4771156](feast-dev@4771156)) * Decouple metrics HTTP server into a dedicated child process ([feast-dev#6855](feast-dev#6855)) ([0b621de](feast-dev@0b621de)) * Reduce online serving hot-path overhead from metrics and audit logging ([5ee7ee3](feast-dev@5ee7ee3)) --------- Signed-off-by: Srihari <svenkata@redhat.com> Signed-off-by: ntkathole <nikhilkathole2683@gmail.com> Co-authored-by: Srihari <svenkata@redhat.com> Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: feast-ci-bot <feast-ci-bot@willem.co>

haoxu0 requested a review from a team as a code owner October 1, 2026 20:27

ntkathole added the ok-to-test label Oct 2, 2026

ntkathole force-pushed the feat/lance-table-format branch from 8d5e3a5 to 9afb378 Compare October 4, 2026 13:05

ntkathole approved these changes Oct 4, 2026

View reviewed changes

ntkathole merged commit 1196e22 into feast-dev:master Oct 4, 2026
21 of 23 checks passed

This was referenced Oct 4, 2026

feat: Lance as an engine-agnostic, catalog-backed, version-pinned data source #6899

Open

feat: Add a non-JVM read path for Lance data sources #6943

Merged

SparkOfflineStore.offline_write_batch ignores table_format and writes with file_format #6955

Open

Navigation Menu

Repository navigation

feat: Add LanceFormat table format with version/tag pinning - #6925

feat: Add LanceFormat table format with version/tag pinning#6925
ntkathole merged 1 commit into
feast-dev:masterfrom
haoxu0:feat/lance-table-format

haoxu0 commented Oct 1, 2026

Uh oh!

codecov-commenter commented Oct 1, 2026 •

edited

Loading

Uh oh!

haoxu0 commented Oct 2, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

haoxu0 commented Oct 1, 2026

Why TableFormat and not a new source

It works with SparkSource unchanged

Pin semantics

Proto regeneration, deliberately constrained

Scope

Testing

Uh oh!

codecov-commenter commented Oct 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

haoxu0 commented Oct 2, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Why `TableFormat` and not a new source

It works with `SparkSource` unchanged

codecov-commenter commented Oct 1, 2026 •

edited

Loading