feat: Add a non-JVM read path for Lance data sources by haoxu0 · Pull Request #6943 · feast-dev/feast

haoxu0 · 2026-10-05T19:29:51Z

Completes the read half of #6899. LanceFormat landed in #6925 as a format descriptor; nothing read Lance until now.

Lance already worked through SparkSource, which drives its reader generically from table_format.format_type.value and table_format.properties. What was missing is a path that needs no JVM — which is also the real test of whether the DataSource abstraction is engine-agnostic rather than Spark-agnostic in name only.

Placement, and why not the two obvious alternatives

A new LanceSource read by the existing DuckDB store — not a Lance offline store, and not an extension of FileSource.

FileSource is the wrong host. Its format axis is already occupied by file_format (ParquetFormat/DeltaFormat), so adding table_format gives one source two overlapping format axes. It is also read by two stores with incompatible contracts: duckdb._read_data_source dispatches on type, while dask._read_datasource has no dispatch seam at all — it reads file_options.uri unconditionally as Parquet and hard-asserts isinstance(..., FileSource) in three places, so a third format would break silently under Dask. Decisively: Lance's catalog addressing (catalog/namespace/table) has no path to put in FileSource.path, so the catalog-based layer would not fit the class even if the path-based one did.

A Lance offline store would be the wrong 120 lines. duckdb.py is not an engine — it is a binding that injects _read_data_source/_write_data_source callbacks into the real engine in ibis.py. A LanceOfflineStore would be a near-verbatim copy of it plus a repo_config.py entry, and would force users to choose Lance instead of Parquet rather than mixing them in one feature service.

Reading a source as ibis.memtable(arrow_table) in the DuckDB store already has two precedents — IcebergSource and MlflowDatasetSource. Following them means ibis.py is untouched, so the point-in-time join, TTL handling, field mapping and ODFVs all work unchanged, and neither repo_config.py nor data_source.py needs an edit because CUSTOM_SOURCE + data_source_class_type is self-describing in the registry proto.

Net: +47 lines to duckdb.py, one new source module.

Both addressing modes work

# path-based
LanceSource(uri="s3://bucket/poi_embeddings.lance", timestamp_field="generated_at")

# catalog-based, through namespace_client + table_id
LanceSource(
    table="poi_embeddings",
    namespace_impl="rest",
    table_format=LanceFormat(catalog="my_catalog", namespace="features"),
    timestamp_field="generated_at",
)

Pin semantics: a pin selects data, never shape

Following what was argued on #5782 and #6925. Three things enforce it:

get_table_column_names_and_types reads the pinned schema, so feast apply infers the schema reads will actually see.
A pre-flight check in get_historical_features fails with a message naming the pin when a pinned version cannot satisfy the declared schema. It reads schemas only — one metadata round trip, no scan.
Vector widths go through _validate_vector_field_lengths from fix: Make vector length validation reachable, schema-driven and vectorized #6909 rather than a second validator. This is exact for Lance because Lance stores vectors as fixed_size_list, where that check is a property of the Arrow type.

Demonstrated rather than asserted — the dataset is overwritten with a corrected value for the same (entity, timestamp), and the pinned read still returns the original:

latest  (        latest version): raw=[0.77, 1.17]  get_historical_features@12:30 -> 0.77
tag pin (    tag 'eval_2026_q1'): raw=[0.5,  0.9 ]  get_historical_features@12:30 -> 0.50
ver pin (             version 1): raw=[0.5,  0.9 ]  get_historical_features@12:30 -> 0.50

Testing

43 tests, all using the dir namespace implementation — it exercises the identical namespace_client + table_id code path as a remote catalog, so the suite needs no server. Point-in-time correctness is covered for both the path-based and catalog-based variants, with a 12:30 query selecting the 11:00 row and never the 13:00 one.

Regression evidence, rebased onto current master. Failure names compared rather than counts, since the suite has flaky members:

with this change : 40 failures
master           : 40 failures
only with change : (none)
only on master   : (none)

ruff check and ruff format --check clean on all four files. mypy --ignore-missing-imports clean. (Bare mypy reports pre-existing missing-stub noise for pyarrow/pandas/ibis/pyiceberg/deltalake; the repo has no mypy config section.)

Known limitations, deliberately

Read-only. _write_data_source is untouched, so a LanceSource is not yet a persist target and there is no SavedDatasetLanceStorage. materialize into an online store does work, via pull_latest_from_table_or_query. Happy to follow up — _write_data_source already dispatches on IcebergSource, so the seam exists.
DuckDB only. Dask cannot read Lance, by the reasoning above. RayOfflineStore would be a natural second host since it already has a _resolve_source_dataset dispatch, but it is not in this PR.
Only the dir impl is exercised. rest, glue, unity, polaris and the Hive/Iceberg impls need lance-namespace-impls plus a live catalog. The code path is identical, which is why dir is a meaningful test, but I have not run against a remote catalog.
Eager materialisation. The read pulls the dataset into Arrow before ibis sees it, matching what IcebergSource and MlflowDatasetSource already do. Column projection is pushed down via to_arrow(columns=...), but the retrieval path does not pass a projection and no predicate pushdown reaches Lance.
Vector-width validation is exact only for fixed_size_list. For a plain list/large_list column the pre-flight runs on a zero-row table and infers nothing. Lance writes fixed_size_list, so this is the right trade in practice. Making it exact in general needs the feature view threaded into the data_source_reader callback in ibis.py, whose signature is (DataSource, str) — a separate change that would benefit every source, not just this one.

Related: #6899, #6925, #6909, #5782, #5652

codecov-commenter · 2026-10-05T19:40:15Z

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

❌ Patch coverage is 1.19760% with 165 lines in your changes missing coverage. Please review.
✅ Project coverage is 48.99%. Comparing base (7087f1f) to head (8b8d17c).
⚠️ Report is 5 commits behind head on master.

Files with missing lines	Patch %	Lines
...t/infra/data_sources/contrib/lance/lance_source.py	0.00%	147 Missing ⚠️
sdk/python/feast/infra/offline_stores/duckdb.py	11.11%	16 Missing ⚠️
...feast/infra/data_sources/contrib/lance/__init__.py	0.00%	2 Missing ⚠️
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #6943      +/-   ##
==========================================
- Coverage   49.08%   48.99%   -0.09%     
==========================================
  Files         433      435       +2     
  Lines       54332    54505     +173     
  Branches     7917     7948      +31     
==========================================
+ Hits        26667    26703      +36     
- Misses      25788    25926     +138     
+ Partials     1877     1876       -1

Flag	Coverage Δ
go-feature-server	`30.58% <ø> (ø)`
python-unit	`50.37% <1.19%> (-0.11%)`	⬇️

Files with missing lines	Coverage Δ
...feast/infra/data_sources/contrib/lance/__init__.py	`0.00% <0.00%> (ø)`
sdk/python/feast/infra/offline_stores/duckdb.py	`33.60% <11.11%> (-1.16%)`	⬇️
...t/infra/data_sources/contrib/lance/lance_source.py	`0.00% <0.00%> (ø)`

... and 6 files with indirect coverage changes

Continue to review full report in Codecov by Harness.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a5eaa36...8b8d17c. Read the comment docs.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Copilot

Copilot review overview

🟡 Changes recommended

Vector schema inference and validation have correctness gaps, and the required Lance dependencies are not installable through a Feast extra.

Review effort: Balanced
Findings: 1 · 3 · 1

Open (5)

Fixed-size Arrow lists fail Feast vector schema inference · New
Missing Lance optional dependency and CI extra · New
Connection credentials are ignored for object-store reads · New
Validate source schema before applying column projections · New
LanceSource constructor does not match documented API · New

What changed in this PR

Adds native, non-JVM Lance reads through the DuckDB offline store, including catalog addressing, version pinning, and schema validation.

Changes:

Introduces LanceSource for URI and namespace-based datasets.
Adds DuckDB dispatch and historical-retrieval validation.
Adds Lance read, pinning, schema, and point-in-time tests.

File	Description
`lance_source.py`	Implements Lance source reads, serialization, and validation.
`lance/__init__.py`	Exports the Lance source API.
`duckdb.py`	Integrates Lance reads with DuckDB retrieval.
`test_lance_source.py`	Tests source behavior and DuckDB integration.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

haoxu0 · 2026-10-06T07:21:44Z

@ntkathole @franciscojavierarceo PTAL when you have time

Adds LanceSource and teaches the DuckDB offline store to read it, completing the read half of feast-dev#6899. LanceFormat landed in feast-dev#6925 as a format descriptor; nothing read Lance until now. Lance already worked through SparkSource, which drives its reader generically from table_format.format_type.value and table_format.properties. What was missing is a path that needs no JVM, which is also the real test of whether the DataSource abstraction is engine-agnostic rather than Spark-agnostic in name only. Placement: a new source read by the existing DuckDB store, rather than a Lance offline store or an extension of FileSource. FileSource is the wrong host. Its format axis is already taken by file_format, so adding table_format would give one source two overlapping format axes. It is also read by two stores with incompatible contracts: duckdb._read_data_source dispatches on type, while dask._read_datasource has no dispatch seam and reads file_options.uri unconditionally as Parquet, and asserts isinstance(..., FileSource) in three places. Decisively, Lance's catalog addressing has no path to put in FileSource.path, so the catalog-based layer would not fit the class even if the path-based one did. A Lance offline store would be the wrong 120 lines. duckdb.py is a binding that injects reader and writer callbacks into the engine in ibis.py, so a Lance store would be a near-copy of it plus a repo_config entry, and would force a choice between Lance and Parquet instead of mixing them in one feature service. Reading a source as ibis.memtable(arrow_table) in the DuckDB store already has two precedents, IcebergSource and MlflowDatasetSource. Following them leaves ibis.py untouched, so the point-in-time join, TTL handling, field mapping and ODFVs work unchanged, and no edit to repo_config.py or data_source.py is needed because CUSTOM_SOURCE plus data_source_class_type is self-describing. Both addressing modes work: a uri, and catalog/namespace/table through namespace_client and table_id. Pin semantics follow what was argued on feast-dev#5782 and feast-dev#6925: a pin selects data, never shape. get_table_column_names_and_types reads the pinned schema so feast apply infers what reads will actually see; a pre-flight check fails with a message naming the pin when a pinned version cannot satisfy the declared schema; and vector widths go through _validate_vector_field_lengths from feast-dev#6909 rather than a second validator. Tests use the dir namespace implementation, which exercises the same namespace_client and table_id code path as a remote catalog with no server required. 43 tests, including a demonstration that a tag pin returns earlier data after the dataset has been overwritten for the same entity and timestamp. Read-only for now: _write_data_source is untouched, so a LanceSource is not yet a persist target and there is no SavedDatasetLanceStorage. Signed-off-by: hao-xu5 <hxu44@apple.com>

Signed-off-by: HaoXuAI <sduxuhao@gmail.com>

feast-dev#6943 made a `LanceSource` readable without Spark. This makes one writable, through the same two addressing modes: a uri, or a `namespace_client` plus `table_id` resolved through a Lance namespace, so a catalog-addressed dataset is committed through its catalog rather than behind its back. `_write_data_source` gains a `LanceSource` branch, following the `IcebergSource` branch already there, which makes the path reachable from `DuckDBOfflineStore.offline_write_batch`. The `isinstance` narrowing the read path open-coded is now a shared `_as_lance_source` helper used by both, rather than a second copy of the optional-import guard. Three things are settled before Lance is called. A pinned source is refused. A Lance commit always produces a new version, so `write_dataset` has no `version` argument at all; a write through a source pinned to version 1 or to a tag would succeed and then be invisible through the very source that performed it. Measured: after appending to a dataset tagged `prod` at version 1, the unpinned source reads 5 rows and the pinned source still reads 3. An absent dataset is created rather than appended to, because Lance has no create-or-append mode and rejects `create` on an existing dataset. An existing dataset's vector widths are compared against the incoming data. Lance rejects an `append` whose schema disagrees, but it accepts an `overwrite` that replaces a 8-wide embedding column with a 32-wide one, silently rewriting the declared shape and leaving a version history whose vectors are not mutually comparable, with every index built on the old width invalidated. No legitimate schema evolution changes an embedding's dimension, so this is refused. The guard is narrow on purpose: only vector widths are policed, and other type changes remain Lance's business. The declared `vector_length` is checked at the store entry point rather than in the writer callback, which is handed a `DataSource` and so cannot see the feature view. It reuses `_validate_vector_field_lengths` from feast-dev#6909 rather than adding a second implementation. Without it, creating a dataset at a width other than the declared one would succeed and fail only on the next read. 19 tests added, all using the `dir` namespace implementation for the catalog-based cases so no server is needed. Signed-off-by: hao-xu5 <hxu44@apple.com>

* feat: Add a non-JVM write path for Lance data sources #6943 made a `LanceSource` readable without Spark. This makes one writable, through the same two addressing modes: a uri, or a `namespace_client` plus `table_id` resolved through a Lance namespace, so a catalog-addressed dataset is committed through its catalog rather than behind its back. `_write_data_source` gains a `LanceSource` branch, following the `IcebergSource` branch already there, which makes the path reachable from `DuckDBOfflineStore.offline_write_batch`. The `isinstance` narrowing the read path open-coded is now a shared `_as_lance_source` helper used by both, rather than a second copy of the optional-import guard. Three things are settled before Lance is called. A pinned source is refused. A Lance commit always produces a new version, so `write_dataset` has no `version` argument at all; a write through a source pinned to version 1 or to a tag would succeed and then be invisible through the very source that performed it. Measured: after appending to a dataset tagged `prod` at version 1, the unpinned source reads 5 rows and the pinned source still reads 3. An absent dataset is created rather than appended to, because Lance has no create-or-append mode and rejects `create` on an existing dataset. An existing dataset's vector widths are compared against the incoming data. Lance rejects an `append` whose schema disagrees, but it accepts an `overwrite` that replaces a 8-wide embedding column with a 32-wide one, silently rewriting the declared shape and leaving a version history whose vectors are not mutually comparable, with every index built on the old width invalidated. No legitimate schema evolution changes an embedding's dimension, so this is refused. The guard is narrow on purpose: only vector widths are policed, and other type changes remain Lance's business. The declared `vector_length` is checked at the store entry point rather than in the writer callback, which is handed a `DataSource` and so cannot see the feature view. It reuses `_validate_vector_field_lengths` from #6909 rather than adding a second implementation. Without it, creating a dataset at a width other than the declared one would succeed and fail only on the next read. 19 tests added, all using the `dir` namespace implementation for the catalog-based cases so no server is needed. Signed-off-by: hao-xu5 <hxu44@apple.com> * fix: Correct what a Lance namespace does during a write The docstring claimed a catalog-addressed write is "committed through its namespace". That is only true of a create. Instrumenting the namespace client shows how far it is actually involved: create -> declare_table append -> describe_table only open -> describe_table, namespace_id So a namespace records that a table exists and where it lives, and does not track its versions: the version an append produces is never reported back to it. Worth stating precisely, because it is the reason the pin contract has to be enforced in `assert_writable` rather than left to the catalog -- no catalog is going to reject a write on a pin's behalf. Adds a test that records the namespace calls, so the claim is pinned by a test rather than asserted in prose. Signed-off-by: hao-xu5 <hxu44@apple.com> * fix: Accept an optional source in _as_lance_source offline_write_batch passes FeatureView.batch_source, which is optional, so the narrow DataSource annotation made mypy reject the call. The body already handled None, since isinstance(None, LanceSource) is False, so only the signature changes. Signed-off-by: hao-xu5 <hxu44@apple.com> --------- Signed-off-by: hao-xu5 <hxu44@apple.com>

# [0.67.0](v0.66.0...v0.67.0) (2026-10-13) ### Bug Fixes * Allow dill 0.4 so UDF serialization works on Python 3.14 ([#6858](#6858)) ([139704f](139704f)) * Allow serving features while a feature view is MATERIALIZING ([#6789](#6789)) ([e2d7fe2](e2d7fe2)), closes [#6780](#6780) * Allowlist the pinned commit SHA flagged by detect-secrets ([4b81d18](4b81d18)) * Apply point-in-time filter correctly with the Polars backend ([#6992](#6992)) ([01224d0](01224d0)) * Authorize registry server writes before deserializing UDFs ([e0ef16e](e0ef16e)) * Automate versioned documentation publication ([#6763](#6763)) ([5ad5592](5ad5592)), closes [#6762](#6762) * Avoid NumPy generic-unit deprecation when importing feast.type_map ([5fea698](5fea698)) * Bind metrics, REST registry, ui, and lineage servers dual-stack ([#6886](#6886)) ([8d2773c](8d2773c)) * Bound Arrow Flight batch size in the offline server ([#6896](#6896)) ([d169238](d169238)) * Compute std and var correctly for tiled aggregations ([213a130](213a130)) * Convert PdfBytes values to protos like ImageBytes ([#6968](#6968)) ([af20fe4](af20fe4)) * Decode fixed_size_list as the matching Feast list value type ([#6944](#6944)) ([8cfe891](8cfe891)) * Do not log /metrics requests from the pre-fork metrics server thread ([7087f1f](7087f1f)), closes [#6928](#6928) * Don't return from finally blocks in REST registry helpers ([d7cb4d0](d7cb4d0)) * Encode non-finite floats in feature server JSON responses ([#6930](#6930)) ([211ecb8](211ecb8)) * Feature freshness logic updated ([#6765](#6765)) ([f0bc070](f0bc070)) * Fix monitoring auth ([015cc84](015cc84)) * Handle non-nanosecond timestamps in tiled aggregations ([b8989cb](b8989cb)) * Improve feature list column sizing ([#6793](#6793)) ([2573693](2573693)) * Include vector attributes in Field equality ([#6934](#6934)) ([298c3f6](298c3f6)) * Increase feast container probe timeouts to avoid false restarts ([e8dbd50](e8dbd50)) * Infer the tensor type from retrieved values in to_tensor ([a03e323](a03e323)) * Infer timestamps for datetime columns in any unit or timezone ([810391f](810391f)) * Infer vector width when vector_length is undeclared ([0056565](0056565)), closes [#6907](#6907) * Initialize auth managers in standalone lineage server ([#6873](#6873)) ([ecd0d38](ecd0d38)) * Keep a zero watermark delay threshold on a Kafka source ([e2e28c7](e2e28c7)) * Keep entity rows apart in the ibis point-in-time join ([a87c070](a87c070)) * Keep the BatchFeatureView type and settings when copying it ([#6989](#6989)) ([013fc57](013fc57)) * Keep tiling, offline and stream engine settings when copying a StreamFeatureView ([fac0f88](fac0f88)) * Limit tiled max and min aggregations to the window ([2583c45](2583c45)) * Load Milvus collections once instead of on every query ([#6882](#6882)) ([f9e91fa](f9e91fa)) * Make vector length validation reachable, schema-driven and vectorized ([#6909](#6909)) ([29af8c0](29af8c0)), closes [#6907](#6907) * Map INT32_LIST, PDF_BYTES and IMAGE_BYTES type names to their ValueType ([8c1e825](8c1e825)) * Map narrow and unsigned Arrow integer types to Feast value types ([#6960](#6960)) ([ae74360](ae74360)) * Map Spark smallint and tinyint columns to INT32 ([9d42729](9d42729)) * Map SQL Server bigint to INT64 and float to DOUBLE ([6d15533](6d15533)) * Map Trino DECIMAL columns to pyarrow decimal128 ([155bc24](155bc24)) * Normalize Arrow-backed dtypes from Ray Dataset.to_pandas() for pyarrow >=25 compatibility ([#6916](#6916)) ([374f1d7](374f1d7)) * **operator:** Set grpc appProtocol on the offline Service port ([f9dcfb5](f9dcfb5)) * Pin expected tensor to same device in online retrieval test ([#6777](#6777)) ([1f2584d](1f2584d)) * Preserve an unset aggregation time window across a proto round-trip ([2000379](2000379)) * Preserve entity row order in precomputed reads ([ce37029](ce37029)) * Pull the MinIO test image from pgsty/minio ([fe27230](fe27230)) * Pull the MinIO test image from quay ([#6838](#6838)) ([367001b](367001b)) * Raise Trino monitoring query errors other than a missing table ([6161766](6161766)) * Read Kafka and Kinesis sources without a batch source from proto ([a620f5c](a620f5c)) * Read naive datetimes as UTC in UnixTimestamp values ([bc703d9](bc703d9)) * Record which storage class wrote a custom_storage payload ([#6988](#6988)) ([c0545f7](c0545f7)) * Refresh the SQL registry cache after deleting a data source ([66c55e3](66c55e3)) * Refresh the SQL registry cache after deleting a permission ([6b212f5](6b212f5)) * Reject unknown resource types and actions in the REST permission API ([a5eaa36](a5eaa36)) * Remove pip install command from Redis GLIDE docstring ([66ff990](66ff990)) * Remove runtime pip install hints from mlflow errors ([dd9f099](dd9f099)) * Remove unsupported vector_len from PostgreSQL tutorial config ([fcc9a85](fcc9a85)) * Repair Chronon operator validation and Pixi CI setup ([#6947](#6947)) ([3710f78](3710f78)) * Report Dask monitoring read errors instead of no data ([81673d9](81673d9)) * Resolve feature view batch source name against registry ([f8883d0](f8883d0)) * Restrict UI server CORS to trusted origins ([#6986](#6986)) ([5e7ef3e](5e7ef3e)) * Return feature_view_metadata from the REST feature server ([#6927](#6927)) ([bc5aeef](bc5aeef)) * Serve aliased on demand feature views in feature services ([823b92e](823b92e)) * Snapshot tag filters in registry cache keys ([81e1546](81e1546)) * Stabilize materialization checks across hour boundaries ([#6948](#6948)) ([acadb17](acadb17)) * Stop the Couchbase online store reporting failed writes as written ([b3f3c98](b3f3c98)) * Support SQL registry dump ([b60f9c7](b60f9c7)) * Treat datetimes with a None utcoffset as naive ([687bee8](687bee8)) * Unwrap SecretStr tokens when building Trino auth objects ([#6771](#6771)) ([922db5c](922db5c)) * Use a valid ShuffleStrategy for the Ray DatasetContext ([6259f5c](6259f5c)) * Use do_exchange for offline server reads to support HPA ([bae07fc](bae07fc)) * Use group_by in Polars backend aggregation ([#6965](#6965)) ([7b9fad2](7b9fad2)) * Use upstream kubectl image as feast-operator CronJob default ([d6cc1dc](d6cc1dc)) * Use UTC end bound and correct docs for disable_event_timestamp ([#6956](#6956)) ([cc07c82](cc07c82)), closes [#6936](#6936) ### Features * Add a non-JVM read path for Lance data sources ([#6943](#6943)) ([e2a53bb](e2a53bb)), closes [#6925](#6925) [#5782](#5782) [#6909](#6909) * Add a non-JVM write path for Lance data sources ([#6945](#6945)) ([f754323](f754323)), closes [#6943](#6943) [#6909](#6909) * Add Chronon online and offline store integrations ([#6188](#6188)) ([dd059c6](dd059c6)) * Add consistency_level to the Milvus online store config ([97296c1](97296c1)) * Add dualStack server option to the Feast operator ([#6887](#6887)) ([725a147](725a147)), closes [#6977](#6977) * Add Lance optional dependencies ([#6958](#6958)) ([20bc993](20bc993)) * Add LanceFormat table format with version/tag pinning ([1196e22](1196e22)), closes [#5650](#5650) * Add MLflow as a first-class offline DataSource for Feast ([#6702](#6702)) ([3667772](3667772)) * Add optional GLIDE client for Redis online reads ([#6870](#6870)) ([6e9a532](6e9a532)) * Add quality monitoring features for Trino offline store ([#6779](#6779)) ([9340545](9340545)) * Add SavedDatasetLanceStorage ([1118d3e](1118d3e)) * Add standalone MCP server as a first-class component ([0efab5c](0efab5c)), closes [python-keycloak#623](https://github.com/python-keycloak/issues/623) * Add structured audit logging for MCP feature server ([#6456](#6456)) ([640ac70](640ac70)) * Add Trino compute engine for batch retrieval and materialization ([#6898](#6898)) ([11a47b4](11a47b4)) * Batch Postgres online reads across feature views ([#6832](#6832)) ([514bee7](514bee7)), closes [#3259](#3259) * **bigtable:** Make write batch size and concurrency configurable ([#6847](#6847)) ([5388522](5388522)) * Bucket feature count labels in latency metrics ([#6811](#6811)) ([d9ea120](d9ea120)) * Expose runtime version in Feast UI ([#6843](#6843)) ([fa8f06b](fa8f06b)) * Make Milvus index and search params configurable ([ef743ff](ef743ff)) * Offline write batch for IcebergSource ([5be5ed0](5be5ed0)) * OpenLineage consumer improvements - dataset versioning, column lineage, ownership ([69f294e](69f294e)) * **operator:** Add tolerations and nodeSelector to the FeatureStore CR ([#6741](#6741)) ([03dac6f](03dac6f)) * Serve the MCP server dual-stack on --host :: ([4fc9726](4fc9726)) * Support Milvus partition keys in the Milvus online store ([0ed3285](0ed3285)) * Support token, uri and db_name in the Milvus online store ([c5efecd](c5efecd)) * Version-pinning for FeatureService (online + offline) ([#6718](#6718)) ([787845a](787845a)) ### Performance Improvements * Avoid pandas copy in Spark reads ([#6878](#6878)) ([4771156](4771156)) * Decouple metrics HTTP server into a dedicated child process ([#6855](#6855)) ([0b621de](0b621de)) * Reduce online serving hot-path overhead from metrics and audit logging ([5ee7ee3](5ee7ee3))

* fix: Increase feast container probe timeouts to avoid false restarts Signed-off-by: Srihari <svenkata@redhat.com> * chore: Remove unused hermeto offline tree and sdist requirement locks (feast-dev#6987) * chore: Remove unused multicloud hermeto offline build tree Drop sdk/python/feast/infra/feature_servers/multicloud/offline/, which only supported local hermeto/arrow-from-source and RPM lock offline image builds and is unused by Makefile CI, GitHub Actions, or the standard feature-server Dockerfiles. Also clear related .secrets.baseline entries. Signed-off-by: ntkathole <nikhilkathole2683@gmail.com> Co-authored-by: Cursor <cursoragent@cursor.com> * chore: Drop unused minimal-sdist requirement locks Remove py3.10–3.12 minimal-sdist(-build) lockfiles, stop generating them in lock-python-dependencies-all, and delete the minimal-sdist-build extra. They only supported hermeto build-from-sdist offline image builds. Signed-off-by: ntkathole <nikhilkathole2683@gmail.com> Co-authored-by: Cursor <cursoragent@cursor.com> * chore: Refresh pixi.lock after removing minimal-sdist-build Keep pixi install --locked in sync with the pyproject.toml optional dependency change so ray/duckdb CI environments can install. Signed-off-by: ntkathole <nikhilkathole2683@gmail.com> Co-authored-by: Cursor <cursoragent@cursor.com> --------- Signed-off-by: ntkathole <nikhilkathole2683@gmail.com> Co-authored-by: Cursor <cursoragent@cursor.com> * chore(release): release 0.67.0 # [0.67.0](feast-dev/feast@v0.66.0...v0.67.0) (2026-10-13) ### Bug Fixes * Allow dill 0.4 so UDF serialization works on Python 3.14 ([feast-dev#6858](feast-dev#6858)) ([139704f](feast-dev@139704f)) * Allow serving features while a feature view is MATERIALIZING ([feast-dev#6789](feast-dev#6789)) ([e2d7fe2](feast-dev@e2d7fe2)), closes [feast-dev#6780](feast-dev#6780) * Allowlist the pinned commit SHA flagged by detect-secrets ([4b81d18](feast-dev@4b81d18)) * Apply point-in-time filter correctly with the Polars backend ([feast-dev#6992](feast-dev#6992)) ([01224d0](feast-dev@01224d0)) * Authorize registry server writes before deserializing UDFs ([e0ef16e](feast-dev@e0ef16e)) * Automate versioned documentation publication ([feast-dev#6763](feast-dev#6763)) ([5ad5592](feast-dev@5ad5592)), closes [feast-dev#6762](feast-dev#6762) * Avoid NumPy generic-unit deprecation when importing feast.type_map ([5fea698](feast-dev@5fea698)) * Bind metrics, REST registry, ui, and lineage servers dual-stack ([feast-dev#6886](feast-dev#6886)) ([8d2773c](feast-dev@8d2773c)) * Bound Arrow Flight batch size in the offline server ([feast-dev#6896](feast-dev#6896)) ([d169238](feast-dev@d169238)) * Compute std and var correctly for tiled aggregations ([213a130](feast-dev@213a130)) * Convert PdfBytes values to protos like ImageBytes ([feast-dev#6968](feast-dev#6968)) ([af20fe4](feast-dev@af20fe4)) * Decode fixed_size_list as the matching Feast list value type ([feast-dev#6944](feast-dev#6944)) ([8cfe891](feast-dev@8cfe891)) * Do not log /metrics requests from the pre-fork metrics server thread ([7087f1f](feast-dev@7087f1f)), closes [feast-dev#6928](feast-dev#6928) * Don't return from finally blocks in REST registry helpers ([d7cb4d0](feast-dev@d7cb4d0)) * Encode non-finite floats in feature server JSON responses ([feast-dev#6930](feast-dev#6930)) ([211ecb8](feast-dev@211ecb8)) * Feature freshness logic updated ([feast-dev#6765](feast-dev#6765)) ([f0bc070](feast-dev@f0bc070)) * Fix monitoring auth ([015cc84](feast-dev@015cc84)) * Handle non-nanosecond timestamps in tiled aggregations ([b8989cb](feast-dev@b8989cb)) * Improve feature list column sizing ([feast-dev#6793](feast-dev#6793)) ([2573693](feast-dev@2573693)) * Include vector attributes in Field equality ([feast-dev#6934](feast-dev#6934)) ([298c3f6](feast-dev@298c3f6)) * Increase feast container probe timeouts to avoid false restarts ([e8dbd50](feast-dev@e8dbd50)) * Infer the tensor type from retrieved values in to_tensor ([a03e323](feast-dev@a03e323)) * Infer timestamps for datetime columns in any unit or timezone ([810391f](feast-dev@810391f)) * Infer vector width when vector_length is undeclared ([0056565](feast-dev@0056565)), closes [feast-dev#6907](feast-dev#6907) * Initialize auth managers in standalone lineage server ([feast-dev#6873](feast-dev#6873)) ([ecd0d38](feast-dev@ecd0d38)) * Keep a zero watermark delay threshold on a Kafka source ([e2e28c7](feast-dev@e2e28c7)) * Keep entity rows apart in the ibis point-in-time join ([a87c070](feast-dev@a87c070)) * Keep the BatchFeatureView type and settings when copying it ([feast-dev#6989](feast-dev#6989)) ([013fc57](feast-dev@013fc57)) * Keep tiling, offline and stream engine settings when copying a StreamFeatureView ([fac0f88](feast-dev@fac0f88)) * Limit tiled max and min aggregations to the window ([2583c45](feast-dev@2583c45)) * Load Milvus collections once instead of on every query ([feast-dev#6882](feast-dev#6882)) ([f9e91fa](feast-dev@f9e91fa)) * Make vector length validation reachable, schema-driven and vectorized ([feast-dev#6909](feast-dev#6909)) ([29af8c0](feast-dev@29af8c0)), closes [feast-dev#6907](feast-dev#6907) * Map INT32_LIST, PDF_BYTES and IMAGE_BYTES type names to their ValueType ([8c1e825](feast-dev@8c1e825)) * Map narrow and unsigned Arrow integer types to Feast value types ([feast-dev#6960](feast-dev#6960)) ([ae74360](feast-dev@ae74360)) * Map Spark smallint and tinyint columns to INT32 ([9d42729](feast-dev@9d42729)) * Map SQL Server bigint to INT64 and float to DOUBLE ([6d15533](feast-dev@6d15533)) * Map Trino DECIMAL columns to pyarrow decimal128 ([155bc24](feast-dev@155bc24)) * Normalize Arrow-backed dtypes from Ray Dataset.to_pandas() for pyarrow >=25 compatibility ([feast-dev#6916](feast-dev#6916)) ([374f1d7](feast-dev@374f1d7)) * **operator:** Set grpc appProtocol on the offline Service port ([f9dcfb5](feast-dev@f9dcfb5)) * Pin expected tensor to same device in online retrieval test ([feast-dev#6777](feast-dev#6777)) ([1f2584d](feast-dev@1f2584d)) * Preserve an unset aggregation time window across a proto round-trip ([2000379](feast-dev@2000379)) * Preserve entity row order in precomputed reads ([ce37029](feast-dev@ce37029)) * Pull the MinIO test image from pgsty/minio ([fe27230](feast-dev@fe27230)) * Pull the MinIO test image from quay ([feast-dev#6838](feast-dev#6838)) ([367001b](feast-dev@367001b)) * Raise Trino monitoring query errors other than a missing table ([6161766](feast-dev@6161766)) * Read Kafka and Kinesis sources without a batch source from proto ([a620f5c](feast-dev@a620f5c)) * Read naive datetimes as UTC in UnixTimestamp values ([bc703d9](feast-dev@bc703d9)) * Record which storage class wrote a custom_storage payload ([feast-dev#6988](feast-dev#6988)) ([c0545f7](feast-dev@c0545f7)) * Refresh the SQL registry cache after deleting a data source ([66c55e3](feast-dev@66c55e3)) * Refresh the SQL registry cache after deleting a permission ([6b212f5](feast-dev@6b212f5)) * Reject unknown resource types and actions in the REST permission API ([a5eaa36](feast-dev@a5eaa36)) * Remove pip install command from Redis GLIDE docstring ([66ff990](feast-dev@66ff990)) * Remove runtime pip install hints from mlflow errors ([dd9f099](feast-dev@dd9f099)) * Remove unsupported vector_len from PostgreSQL tutorial config ([fcc9a85](feast-dev@fcc9a85)) * Repair Chronon operator validation and Pixi CI setup ([feast-dev#6947](feast-dev#6947)) ([3710f78](feast-dev@3710f78)) * Report Dask monitoring read errors instead of no data ([81673d9](feast-dev@81673d9)) * Resolve feature view batch source name against registry ([f8883d0](feast-dev@f8883d0)) * Restrict UI server CORS to trusted origins ([feast-dev#6986](feast-dev#6986)) ([5e7ef3e](feast-dev@5e7ef3e)) * Return feature_view_metadata from the REST feature server ([feast-dev#6927](feast-dev#6927)) ([bc5aeef](feast-dev@bc5aeef)) * Serve aliased on demand feature views in feature services ([823b92e](feast-dev@823b92e)) * Snapshot tag filters in registry cache keys ([81e1546](feast-dev@81e1546)) * Stabilize materialization checks across hour boundaries ([feast-dev#6948](feast-dev#6948)) ([acadb17](feast-dev@acadb17)) * Stop the Couchbase online store reporting failed writes as written ([b3f3c98](feast-dev@b3f3c98)) * Support SQL registry dump ([b60f9c7](feast-dev@b60f9c7)) * Treat datetimes with a None utcoffset as naive ([687bee8](feast-dev@687bee8)) * Unwrap SecretStr tokens when building Trino auth objects ([feast-dev#6771](feast-dev#6771)) ([922db5c](feast-dev@922db5c)) * Use a valid ShuffleStrategy for the Ray DatasetContext ([6259f5c](feast-dev@6259f5c)) * Use do_exchange for offline server reads to support HPA ([bae07fc](feast-dev@bae07fc)) * Use group_by in Polars backend aggregation ([feast-dev#6965](feast-dev#6965)) ([7b9fad2](feast-dev@7b9fad2)) * Use upstream kubectl image as feast-operator CronJob default ([d6cc1dc](feast-dev@d6cc1dc)) * Use UTC end bound and correct docs for disable_event_timestamp ([feast-dev#6956](feast-dev#6956)) ([cc07c82](feast-dev@cc07c82)), closes [feast-dev#6936](feast-dev#6936) ### Features * Add a non-JVM read path for Lance data sources ([feast-dev#6943](feast-dev#6943)) ([e2a53bb](feast-dev@e2a53bb)), closes [feast-dev#6925](feast-dev#6925) [feast-dev#5782](feast-dev#5782) [feast-dev#6909](feast-dev#6909) * Add a non-JVM write path for Lance data sources ([feast-dev#6945](feast-dev#6945)) ([f754323](feast-dev@f754323)), closes [feast-dev#6943](feast-dev#6943) [feast-dev#6909](feast-dev#6909) * Add Chronon online and offline store integrations ([feast-dev#6188](feast-dev#6188)) ([dd059c6](feast-dev@dd059c6)) * Add consistency_level to the Milvus online store config ([97296c1](feast-dev@97296c1)) * Add dualStack server option to the Feast operator ([feast-dev#6887](feast-dev#6887)) ([725a147](feast-dev@725a147)), closes [feast-dev#6977](feast-dev#6977) * Add Lance optional dependencies ([feast-dev#6958](feast-dev#6958)) ([20bc993](feast-dev@20bc993)) * Add LanceFormat table format with version/tag pinning ([1196e22](feast-dev@1196e22)), closes [feast-dev#5650](feast-dev#5650) * Add MLflow as a first-class offline DataSource for Feast ([feast-dev#6702](feast-dev#6702)) ([3667772](feast-dev@3667772)) * Add optional GLIDE client for Redis online reads ([feast-dev#6870](feast-dev#6870)) ([6e9a532](feast-dev@6e9a532)) * Add quality monitoring features for Trino offline store ([feast-dev#6779](feast-dev#6779)) ([9340545](feast-dev@9340545)) * Add SavedDatasetLanceStorage ([1118d3e](feast-dev@1118d3e)) * Add standalone MCP server as a first-class component ([0efab5c](feast-dev@0efab5c)), closes [python-keycloak#623](https://github.com/python-keycloak/issues/623) * Add structured audit logging for MCP feature server ([feast-dev#6456](feast-dev#6456)) ([640ac70](feast-dev@640ac70)) * Add Trino compute engine for batch retrieval and materialization ([feast-dev#6898](feast-dev#6898)) ([11a47b4](feast-dev@11a47b4)) * Batch Postgres online reads across feature views ([feast-dev#6832](feast-dev#6832)) ([514bee7](feast-dev@514bee7)), closes [feast-dev#3259](feast-dev#3259) * **bigtable:** Make write batch size and concurrency configurable ([feast-dev#6847](feast-dev#6847)) ([5388522](feast-dev@5388522)) * Bucket feature count labels in latency metrics ([feast-dev#6811](feast-dev#6811)) ([d9ea120](feast-dev@d9ea120)) * Expose runtime version in Feast UI ([feast-dev#6843](feast-dev#6843)) ([fa8f06b](feast-dev@fa8f06b)) * Make Milvus index and search params configurable ([ef743ff](feast-dev@ef743ff)) * Offline write batch for IcebergSource ([5be5ed0](feast-dev@5be5ed0)) * OpenLineage consumer improvements - dataset versioning, column lineage, ownership ([69f294e](feast-dev@69f294e)) * **operator:** Add tolerations and nodeSelector to the FeatureStore CR ([feast-dev#6741](feast-dev#6741)) ([03dac6f](feast-dev@03dac6f)) * Serve the MCP server dual-stack on --host :: ([4fc9726](feast-dev@4fc9726)) * Support Milvus partition keys in the Milvus online store ([0ed3285](feast-dev@0ed3285)) * Support token, uri and db_name in the Milvus online store ([c5efecd](feast-dev@c5efecd)) * Version-pinning for FeatureService (online + offline) ([feast-dev#6718](feast-dev#6718)) ([787845a](feast-dev@787845a)) ### Performance Improvements * Avoid pandas copy in Spark reads ([feast-dev#6878](feast-dev#6878)) ([4771156](feast-dev@4771156)) * Decouple metrics HTTP server into a dedicated child process ([feast-dev#6855](feast-dev#6855)) ([0b621de](feast-dev@0b621de)) * Reduce online serving hot-path overhead from metrics and audit logging ([5ee7ee3](feast-dev@5ee7ee3)) --------- Signed-off-by: Srihari <svenkata@redhat.com> Signed-off-by: ntkathole <nikhilkathole2683@gmail.com> Co-authored-by: Srihari <svenkata@redhat.com> Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: feast-ci-bot <feast-ci-bot@willem.co>

haoxu0 requested a review from a team as a code owner October 5, 2026 19:29

haoxu0 mentioned this pull request Oct 5, 2026

feat: Add a non-JVM write path for Lance data sources #6945

Merged

haoxu0 added the ok-to-test label Oct 5, 2026

haoxu0 requested review from a team, ejscribner, nquinn408 and robhowley and a balanced review from Copilot and removed request for a team October 6, 2026 06:55

Copilot started reviewing on behalf of haoxu0 October 6, 2026 06:55

View session

haoxu0 requested a review from franciscojavierarceo October 6, 2026 06:55

Copilot AI reviewed Oct 6, 2026

View reviewed changes

ntkathole reviewed Oct 7, 2026

View reviewed changes

Comment thread sdk/python/feast/infra/data_sources/contrib/lance/lance_source.py Outdated

ntkathole reviewed Oct 7, 2026

View reviewed changes

Comment thread sdk/python/feast/infra/data_sources/contrib/lance/lance_source.py

ntkathole approved these changes Oct 7, 2026

View reviewed changes

haoxu0 mentioned this pull request Oct 7, 2026

feat: Add Lance optional dependencies #6958

Merged

haoxu0 and others added 2 commits October 7, 2026 11:19

fix: Address Lance read review feedback

8b8d17c

Signed-off-by: HaoXuAI <sduxuhao@gmail.com>

ntkathole force-pushed the feat/lance-read branch from d57bcdf to 8b8d17c Compare October 7, 2026 05:49

ntkathole merged commit e2a53bb into feast-dev:master Oct 7, 2026
18 of 23 checks passed

rhods-devops-app Bot mentioned this pull request Oct 14, 2026

Automated Sync from main to stable red-hat-data-services/feast#493

Merged

Navigation Menu

Repository navigation

feat: Add a non-JVM read path for Lance data sources - #6943

feat: Add a non-JVM read path for Lance data sources#6943
ntkathole merged 2 commits into
feast-dev:masterfrom
haoxu0:feat/lance-read

haoxu0 commented Oct 5, 2026 •

edited

Loading

Uh oh!

codecov-commenter commented Oct 5, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

haoxu0 commented Oct 6, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

haoxu0 commented Oct 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Placement, and why not the two obvious alternatives

Both addressing modes work

Pin semantics: a pin selects data, never shape

Testing

Known limitations, deliberately

Uh oh!

codecov-commenter commented Oct 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

haoxu0 commented Oct 6, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

haoxu0 commented Oct 5, 2026 •

edited

Loading

codecov-commenter commented Oct 5, 2026 •

edited

Loading