feat: Add a non-JVM write path for Lance data sources by haoxu0 · Pull Request #6945 · feast-dev/feast

haoxu0 · 2026-10-05T20:31:04Z

Rebased onto master now that #6943 (the read path) and #6944 (fixed_size_list decoding) have merged. The two commits that carried them are gone; what is left is the write path alone, and it applies cleanly.

What this does

#6943 made a LanceSource readable without Spark. This makes one writable, through the same two addressing modes — a uri, or a namespace_client + table_id resolved through a Lance namespace — so a catalog-addressed dataset is written to the location its namespace resolves rather than to one assembled locally.

_write_data_source gains a LanceSource branch, following the IcebergSource branch already there, which makes the path reachable from DuckDBOfflineStore.offline_write_batch. The isinstance narrowing that the read path open-coded is now a shared _as_lance_source helper used by both, rather than a second copy of the optional-import guard.

+94 lines in duckdb.py, +121 in lance_source.py.

How far the namespace is involved

Measured by recording the calls a write makes on the namespace client:

operation	namespace calls
create	`declare_table`
append	`describe_table` only
open	`describe_table`, `namespace_id`

So a Lance namespace records that a table exists and where it lives, and does not track its versions — the version an append produces is never reported back to it. This is the reason the pin contract below has to be enforced locally: no catalog is going to reject a write on a pin's behalf. There is a test that records these calls so the behaviour is pinned rather than assumed.

Three things settled before Lance is called

A pinned source is refused. A Lance commit always produces a new version, so write_dataset has no version argument at all. A write through a source pinned to version 1 or to a tag would succeed and then be invisible through the very source that performed it. Measured:

tag prod -> version 1
appended; latest ver 2 | tag prod still reads rows: 10   <-- write invisible to pinned source

An absent dataset is created rather than appended to, because Lance has no create-or-append mode:

CREATE on existing: REJECTED -> OSError Dataset already exists

An existing dataset's vector widths are compared against the incoming data. This is the one case Lance itself permits silently:

APPEND dim-mismatch:  REJECTED -> `emb` should have type fixed_size_list:float:8 but type was fixed_size_list:float:16
OVERWRITE 8->32:      fixed_size_list<item: float>[32]  ver 3    <-- accepted, schema silently rewritten
  v1 still readable:  fixed_size_list<item: float>[8]            <-- history now self-inconsistent

So a single overwrite can leave a dataset whose version history has mutually incomparable vectors, with every ANN index built on the old width invalidated. No legitimate schema evolution changes an embedding's dimension, so this is refused rather than carried out.

The guard is narrow on purpose: only vector widths are policed. Other type changes stay Lance's business, because for those an overwrite is a defensible evolution — there is a test asserting an int64 → string overwrite still goes through.

Declared vector_length

Checked at the store entry point, not in the writer callback, which is handed a DataSource and so cannot see the feature view. It reuses _validate_vector_field_lengths from #6909 rather than adding a second implementation. Without it, creating a dataset at a width other than the declared one would succeed and fail only on the next read.

Mode semantics

entry point	mode	passed by
`offline_write_batch`	`append`	`offline_write_batch_ibis`, 3 positional args
`persist`	`overwrite`	`ibis.py` `persist`

overwrite without allow_overwrite raises SavedDatasetLocationAlreadyExists, matching the FileSource branch.

persist is not reachable for Lance yet: it needs a SavedDatasetStorage, and _DATA_SOURCE_TO_SAVED_DATASET_STORAGE currently maps only FileSource and SparkSource (IcebergSource has no entry either). I left that for a follow-up because the obvious slot is a problem on its own — _proto_attr_name = "custom_storage" is already claimed by three classes (couchbase, clickhouse, postgres) against a plain-dict registry, so SavedDatasetStorage.from_proto dispatches to whichever was imported last. I raised that as #6946 rather than add a fourth claimant. overwrite is still implemented and tested directly, since it is required for correctness once that lands.

A Spark write path is also deliberately out of scope here. SparkOfflineStore.offline_write_batch consults only file_format and ignores table_format, which I raised as #6955; that needs a decision about which of the two axes is authoritative before a Lance branch there would mean anything.

Verification

20 tests added; 65 pass in test_lance_source.py (45 from #6943 plus these). Catalog-based cases use the dir namespace implementation, which drives the identical namespace_client + table_id code path as a remote catalog, so no server is needed.

Covered: create and append for both addressing modes; version-pin and tag-pin refusal; a refused write leaving the dataset at its original version and row count; the append-invisible-to-an-earlier-pin property that motivates the refusal; overwrite with and without allow_overwrite; vector-width change refused and width-preserving overwrite allowed; a non-vector type change left alone; a width guard ignoring columns absent from the write; declared-width enforcement at offline_write_batch; the namespace call sequence above; and a write-then-read round trip through _read_data_source.

Full sdk/python/tests/unit: 40 failures on master and 53 on this branch, but the 13 names in the difference all reproduce on untouched master when that subset is run in isolation — they are pre-existing order- and environment-dependent cases in test_milvus_online_store, test_chronon_online_store and test_metrics, not regressions. Nothing in the difference touches Lance.

ruff check sdk/python/ and ruff format --check clean.

One caveat worth stating plainly: as Copilot noted on #6943, this test module is skipped entirely by its module-level importorskip, so these tests do not run in CI today. #6958 adds the lance extra and wires it into feast[ci], which is what will actually turn them on. Until it lands, the numbers above are local.

codecov-commenter · 2026-10-05T20:46:17Z

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

❌ Patch coverage is 3.03030% with 64 lines in your changes missing coverage. Please review.
✅ Project coverage is 49.44%. Comparing base (87ef218) to head (ee23843).
⚠️ Report is 2 commits behind head on master.

Files with missing lines	Patch %	Lines
...t/infra/data_sources/contrib/lance/lance_source.py	0.00%	39 Missing ⚠️
sdk/python/feast/infra/offline_stores/duckdb.py	7.40%	25 Missing ⚠️
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #6945      +/-   ##
==========================================
- Coverage   49.49%   49.44%   -0.05%     
==========================================
  Files         443      443              
  Lines       55451    55511      +60     
  Branches     8085     8096      +11     
==========================================
+ Hits        27443    27445       +2     
- Misses      26110    26168      +58     
  Partials     1898     1898

Flag	Coverage Δ		*Carryforward flag
go-feature-server	`30.58% <ø> (ø)`		Carriedforward from 87ef218
python-unit	`50.83% <3.03%> (-0.06%)`	⬇️

*This pull request uses carry forward flags. Click here to find out more.

Files with missing lines	Coverage Δ
sdk/python/feast/infra/offline_stores/duckdb.py	`32.30% <7.40%> (-1.30%)`	⬇️
...t/infra/data_sources/contrib/lance/lance_source.py	`0.00% <0.00%> (ø)`

Continue to review full report in Codecov by Harness.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c005dc3...ee23843. Read the comment docs.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

haoxu0 · 2026-10-09T01:56:13Z

@ntkathole PTAL thanks!

ntkathole · 2026-10-09T03:10:02Z

+    data_source.assert_writable()
+
+    arrow_table = table.to_pyarrow()
+    existing_schema = data_source.get_existing_schema()


get_existing_schema() opens the dataset for its schema, then write_dataset opens it again. Negligible for local, one extra round trip for a remote catalog. Fine as is.

feast-dev#6943 made a `LanceSource` readable without Spark. This makes one writable, through the same two addressing modes: a uri, or a `namespace_client` plus `table_id` resolved through a Lance namespace, so a catalog-addressed dataset is committed through its catalog rather than behind its back. `_write_data_source` gains a `LanceSource` branch, following the `IcebergSource` branch already there, which makes the path reachable from `DuckDBOfflineStore.offline_write_batch`. The `isinstance` narrowing the read path open-coded is now a shared `_as_lance_source` helper used by both, rather than a second copy of the optional-import guard. Three things are settled before Lance is called. A pinned source is refused. A Lance commit always produces a new version, so `write_dataset` has no `version` argument at all; a write through a source pinned to version 1 or to a tag would succeed and then be invisible through the very source that performed it. Measured: after appending to a dataset tagged `prod` at version 1, the unpinned source reads 5 rows and the pinned source still reads 3. An absent dataset is created rather than appended to, because Lance has no create-or-append mode and rejects `create` on an existing dataset. An existing dataset's vector widths are compared against the incoming data. Lance rejects an `append` whose schema disagrees, but it accepts an `overwrite` that replaces a 8-wide embedding column with a 32-wide one, silently rewriting the declared shape and leaving a version history whose vectors are not mutually comparable, with every index built on the old width invalidated. No legitimate schema evolution changes an embedding's dimension, so this is refused. The guard is narrow on purpose: only vector widths are policed, and other type changes remain Lance's business. The declared `vector_length` is checked at the store entry point rather than in the writer callback, which is handed a `DataSource` and so cannot see the feature view. It reuses `_validate_vector_field_lengths` from feast-dev#6909 rather than adding a second implementation. Without it, creating a dataset at a width other than the declared one would succeed and fail only on the next read. 19 tests added, all using the `dir` namespace implementation for the catalog-based cases so no server is needed. Signed-off-by: hao-xu5 <hxu44@apple.com>

The docstring claimed a catalog-addressed write is "committed through its namespace". That is only true of a create. Instrumenting the namespace client shows how far it is actually involved: create -> declare_table append -> describe_table only open -> describe_table, namespace_id So a namespace records that a table exists and where it lives, and does not track its versions: the version an append produces is never reported back to it. Worth stating precisely, because it is the reason the pin contract has to be enforced in `assert_writable` rather than left to the catalog -- no catalog is going to reject a write on a pin's behalf. Adds a test that records the namespace calls, so the claim is pinned by a test rather than asserted in prose. Signed-off-by: hao-xu5 <hxu44@apple.com>

offline_write_batch passes FeatureView.batch_source, which is optional, so the narrow DataSource annotation made mypy reject the call. The body already handled None, since isinstance(None, LanceSource) is False, so only the signature changes. Signed-off-by: hao-xu5 <hxu44@apple.com>

# [0.67.0](v0.66.0...v0.67.0) (2026-10-13) ### Bug Fixes * Allow dill 0.4 so UDF serialization works on Python 3.14 ([#6858](#6858)) ([139704f](139704f)) * Allow serving features while a feature view is MATERIALIZING ([#6789](#6789)) ([e2d7fe2](e2d7fe2)), closes [#6780](#6780) * Allowlist the pinned commit SHA flagged by detect-secrets ([4b81d18](4b81d18)) * Apply point-in-time filter correctly with the Polars backend ([#6992](#6992)) ([01224d0](01224d0)) * Authorize registry server writes before deserializing UDFs ([e0ef16e](e0ef16e)) * Automate versioned documentation publication ([#6763](#6763)) ([5ad5592](5ad5592)), closes [#6762](#6762) * Avoid NumPy generic-unit deprecation when importing feast.type_map ([5fea698](5fea698)) * Bind metrics, REST registry, ui, and lineage servers dual-stack ([#6886](#6886)) ([8d2773c](8d2773c)) * Bound Arrow Flight batch size in the offline server ([#6896](#6896)) ([d169238](d169238)) * Compute std and var correctly for tiled aggregations ([213a130](213a130)) * Convert PdfBytes values to protos like ImageBytes ([#6968](#6968)) ([af20fe4](af20fe4)) * Decode fixed_size_list as the matching Feast list value type ([#6944](#6944)) ([8cfe891](8cfe891)) * Do not log /metrics requests from the pre-fork metrics server thread ([7087f1f](7087f1f)), closes [#6928](#6928) * Don't return from finally blocks in REST registry helpers ([d7cb4d0](d7cb4d0)) * Encode non-finite floats in feature server JSON responses ([#6930](#6930)) ([211ecb8](211ecb8)) * Feature freshness logic updated ([#6765](#6765)) ([f0bc070](f0bc070)) * Fix monitoring auth ([015cc84](015cc84)) * Handle non-nanosecond timestamps in tiled aggregations ([b8989cb](b8989cb)) * Improve feature list column sizing ([#6793](#6793)) ([2573693](2573693)) * Include vector attributes in Field equality ([#6934](#6934)) ([298c3f6](298c3f6)) * Increase feast container probe timeouts to avoid false restarts ([e8dbd50](e8dbd50)) * Infer the tensor type from retrieved values in to_tensor ([a03e323](a03e323)) * Infer timestamps for datetime columns in any unit or timezone ([810391f](810391f)) * Infer vector width when vector_length is undeclared ([0056565](0056565)), closes [#6907](#6907) * Initialize auth managers in standalone lineage server ([#6873](#6873)) ([ecd0d38](ecd0d38)) * Keep a zero watermark delay threshold on a Kafka source ([e2e28c7](e2e28c7)) * Keep entity rows apart in the ibis point-in-time join ([a87c070](a87c070)) * Keep the BatchFeatureView type and settings when copying it ([#6989](#6989)) ([013fc57](013fc57)) * Keep tiling, offline and stream engine settings when copying a StreamFeatureView ([fac0f88](fac0f88)) * Limit tiled max and min aggregations to the window ([2583c45](2583c45)) * Load Milvus collections once instead of on every query ([#6882](#6882)) ([f9e91fa](f9e91fa)) * Make vector length validation reachable, schema-driven and vectorized ([#6909](#6909)) ([29af8c0](29af8c0)), closes [#6907](#6907) * Map INT32_LIST, PDF_BYTES and IMAGE_BYTES type names to their ValueType ([8c1e825](8c1e825)) * Map narrow and unsigned Arrow integer types to Feast value types ([#6960](#6960)) ([ae74360](ae74360)) * Map Spark smallint and tinyint columns to INT32 ([9d42729](9d42729)) * Map SQL Server bigint to INT64 and float to DOUBLE ([6d15533](6d15533)) * Map Trino DECIMAL columns to pyarrow decimal128 ([155bc24](155bc24)) * Normalize Arrow-backed dtypes from Ray Dataset.to_pandas() for pyarrow >=25 compatibility ([#6916](#6916)) ([374f1d7](374f1d7)) * **operator:** Set grpc appProtocol on the offline Service port ([f9dcfb5](f9dcfb5)) * Pin expected tensor to same device in online retrieval test ([#6777](#6777)) ([1f2584d](1f2584d)) * Preserve an unset aggregation time window across a proto round-trip ([2000379](2000379)) * Preserve entity row order in precomputed reads ([ce37029](ce37029)) * Pull the MinIO test image from pgsty/minio ([fe27230](fe27230)) * Pull the MinIO test image from quay ([#6838](#6838)) ([367001b](367001b)) * Raise Trino monitoring query errors other than a missing table ([6161766](6161766)) * Read Kafka and Kinesis sources without a batch source from proto ([a620f5c](a620f5c)) * Read naive datetimes as UTC in UnixTimestamp values ([bc703d9](bc703d9)) * Record which storage class wrote a custom_storage payload ([#6988](#6988)) ([c0545f7](c0545f7)) * Refresh the SQL registry cache after deleting a data source ([66c55e3](66c55e3)) * Refresh the SQL registry cache after deleting a permission ([6b212f5](6b212f5)) * Reject unknown resource types and actions in the REST permission API ([a5eaa36](a5eaa36)) * Remove pip install command from Redis GLIDE docstring ([66ff990](66ff990)) * Remove runtime pip install hints from mlflow errors ([dd9f099](dd9f099)) * Remove unsupported vector_len from PostgreSQL tutorial config ([fcc9a85](fcc9a85)) * Repair Chronon operator validation and Pixi CI setup ([#6947](#6947)) ([3710f78](3710f78)) * Report Dask monitoring read errors instead of no data ([81673d9](81673d9)) * Resolve feature view batch source name against registry ([f8883d0](f8883d0)) * Restrict UI server CORS to trusted origins ([#6986](#6986)) ([5e7ef3e](5e7ef3e)) * Return feature_view_metadata from the REST feature server ([#6927](#6927)) ([bc5aeef](bc5aeef)) * Serve aliased on demand feature views in feature services ([823b92e](823b92e)) * Snapshot tag filters in registry cache keys ([81e1546](81e1546)) * Stabilize materialization checks across hour boundaries ([#6948](#6948)) ([acadb17](acadb17)) * Stop the Couchbase online store reporting failed writes as written ([b3f3c98](b3f3c98)) * Support SQL registry dump ([b60f9c7](b60f9c7)) * Treat datetimes with a None utcoffset as naive ([687bee8](687bee8)) * Unwrap SecretStr tokens when building Trino auth objects ([#6771](#6771)) ([922db5c](922db5c)) * Use a valid ShuffleStrategy for the Ray DatasetContext ([6259f5c](6259f5c)) * Use do_exchange for offline server reads to support HPA ([bae07fc](bae07fc)) * Use group_by in Polars backend aggregation ([#6965](#6965)) ([7b9fad2](7b9fad2)) * Use upstream kubectl image as feast-operator CronJob default ([d6cc1dc](d6cc1dc)) * Use UTC end bound and correct docs for disable_event_timestamp ([#6956](#6956)) ([cc07c82](cc07c82)), closes [#6936](#6936) ### Features * Add a non-JVM read path for Lance data sources ([#6943](#6943)) ([e2a53bb](e2a53bb)), closes [#6925](#6925) [#5782](#5782) [#6909](#6909) * Add a non-JVM write path for Lance data sources ([#6945](#6945)) ([f754323](f754323)), closes [#6943](#6943) [#6909](#6909) * Add Chronon online and offline store integrations ([#6188](#6188)) ([dd059c6](dd059c6)) * Add consistency_level to the Milvus online store config ([97296c1](97296c1)) * Add dualStack server option to the Feast operator ([#6887](#6887)) ([725a147](725a147)), closes [#6977](#6977) * Add Lance optional dependencies ([#6958](#6958)) ([20bc993](20bc993)) * Add LanceFormat table format with version/tag pinning ([1196e22](1196e22)), closes [#5650](#5650) * Add MLflow as a first-class offline DataSource for Feast ([#6702](#6702)) ([3667772](3667772)) * Add optional GLIDE client for Redis online reads ([#6870](#6870)) ([6e9a532](6e9a532)) * Add quality monitoring features for Trino offline store ([#6779](#6779)) ([9340545](9340545)) * Add SavedDatasetLanceStorage ([1118d3e](1118d3e)) * Add standalone MCP server as a first-class component ([0efab5c](0efab5c)), closes [python-keycloak#623](https://github.com/python-keycloak/issues/623) * Add structured audit logging for MCP feature server ([#6456](#6456)) ([640ac70](640ac70)) * Add Trino compute engine for batch retrieval and materialization ([#6898](#6898)) ([11a47b4](11a47b4)) * Batch Postgres online reads across feature views ([#6832](#6832)) ([514bee7](514bee7)), closes [#3259](#3259) * **bigtable:** Make write batch size and concurrency configurable ([#6847](#6847)) ([5388522](5388522)) * Bucket feature count labels in latency metrics ([#6811](#6811)) ([d9ea120](d9ea120)) * Expose runtime version in Feast UI ([#6843](#6843)) ([fa8f06b](fa8f06b)) * Make Milvus index and search params configurable ([ef743ff](ef743ff)) * Offline write batch for IcebergSource ([5be5ed0](5be5ed0)) * OpenLineage consumer improvements - dataset versioning, column lineage, ownership ([69f294e](69f294e)) * **operator:** Add tolerations and nodeSelector to the FeatureStore CR ([#6741](#6741)) ([03dac6f](03dac6f)) * Serve the MCP server dual-stack on --host :: ([4fc9726](4fc9726)) * Support Milvus partition keys in the Milvus online store ([0ed3285](0ed3285)) * Support token, uri and db_name in the Milvus online store ([c5efecd](c5efecd)) * Version-pinning for FeatureService (online + offline) ([#6718](#6718)) ([787845a](787845a)) ### Performance Improvements * Avoid pandas copy in Spark reads ([#6878](#6878)) ([4771156](4771156)) * Decouple metrics HTTP server into a dedicated child process ([#6855](#6855)) ([0b621de](0b621de)) * Reduce online serving hot-path overhead from metrics and audit logging ([5ee7ee3](5ee7ee3))

* fix: Increase feast container probe timeouts to avoid false restarts Signed-off-by: Srihari <svenkata@redhat.com> * chore: Remove unused hermeto offline tree and sdist requirement locks (feast-dev#6987) * chore: Remove unused multicloud hermeto offline build tree Drop sdk/python/feast/infra/feature_servers/multicloud/offline/, which only supported local hermeto/arrow-from-source and RPM lock offline image builds and is unused by Makefile CI, GitHub Actions, or the standard feature-server Dockerfiles. Also clear related .secrets.baseline entries. Signed-off-by: ntkathole <nikhilkathole2683@gmail.com> Co-authored-by: Cursor <cursoragent@cursor.com> * chore: Drop unused minimal-sdist requirement locks Remove py3.10–3.12 minimal-sdist(-build) lockfiles, stop generating them in lock-python-dependencies-all, and delete the minimal-sdist-build extra. They only supported hermeto build-from-sdist offline image builds. Signed-off-by: ntkathole <nikhilkathole2683@gmail.com> Co-authored-by: Cursor <cursoragent@cursor.com> * chore: Refresh pixi.lock after removing minimal-sdist-build Keep pixi install --locked in sync with the pyproject.toml optional dependency change so ray/duckdb CI environments can install. Signed-off-by: ntkathole <nikhilkathole2683@gmail.com> Co-authored-by: Cursor <cursoragent@cursor.com> --------- Signed-off-by: ntkathole <nikhilkathole2683@gmail.com> Co-authored-by: Cursor <cursoragent@cursor.com> * chore(release): release 0.67.0 # [0.67.0](feast-dev/feast@v0.66.0...v0.67.0) (2026-10-13) ### Bug Fixes * Allow dill 0.4 so UDF serialization works on Python 3.14 ([feast-dev#6858](feast-dev#6858)) ([139704f](feast-dev@139704f)) * Allow serving features while a feature view is MATERIALIZING ([feast-dev#6789](feast-dev#6789)) ([e2d7fe2](feast-dev@e2d7fe2)), closes [feast-dev#6780](feast-dev#6780) * Allowlist the pinned commit SHA flagged by detect-secrets ([4b81d18](feast-dev@4b81d18)) * Apply point-in-time filter correctly with the Polars backend ([feast-dev#6992](feast-dev#6992)) ([01224d0](feast-dev@01224d0)) * Authorize registry server writes before deserializing UDFs ([e0ef16e](feast-dev@e0ef16e)) * Automate versioned documentation publication ([feast-dev#6763](feast-dev#6763)) ([5ad5592](feast-dev@5ad5592)), closes [feast-dev#6762](feast-dev#6762) * Avoid NumPy generic-unit deprecation when importing feast.type_map ([5fea698](feast-dev@5fea698)) * Bind metrics, REST registry, ui, and lineage servers dual-stack ([feast-dev#6886](feast-dev#6886)) ([8d2773c](feast-dev@8d2773c)) * Bound Arrow Flight batch size in the offline server ([feast-dev#6896](feast-dev#6896)) ([d169238](feast-dev@d169238)) * Compute std and var correctly for tiled aggregations ([213a130](feast-dev@213a130)) * Convert PdfBytes values to protos like ImageBytes ([feast-dev#6968](feast-dev#6968)) ([af20fe4](feast-dev@af20fe4)) * Decode fixed_size_list as the matching Feast list value type ([feast-dev#6944](feast-dev#6944)) ([8cfe891](feast-dev@8cfe891)) * Do not log /metrics requests from the pre-fork metrics server thread ([7087f1f](feast-dev@7087f1f)), closes [feast-dev#6928](feast-dev#6928) * Don't return from finally blocks in REST registry helpers ([d7cb4d0](feast-dev@d7cb4d0)) * Encode non-finite floats in feature server JSON responses ([feast-dev#6930](feast-dev#6930)) ([211ecb8](feast-dev@211ecb8)) * Feature freshness logic updated ([feast-dev#6765](feast-dev#6765)) ([f0bc070](feast-dev@f0bc070)) * Fix monitoring auth ([015cc84](feast-dev@015cc84)) * Handle non-nanosecond timestamps in tiled aggregations ([b8989cb](feast-dev@b8989cb)) * Improve feature list column sizing ([feast-dev#6793](feast-dev#6793)) ([2573693](feast-dev@2573693)) * Include vector attributes in Field equality ([feast-dev#6934](feast-dev#6934)) ([298c3f6](feast-dev@298c3f6)) * Increase feast container probe timeouts to avoid false restarts ([e8dbd50](feast-dev@e8dbd50)) * Infer the tensor type from retrieved values in to_tensor ([a03e323](feast-dev@a03e323)) * Infer timestamps for datetime columns in any unit or timezone ([810391f](feast-dev@810391f)) * Infer vector width when vector_length is undeclared ([0056565](feast-dev@0056565)), closes [feast-dev#6907](feast-dev#6907) * Initialize auth managers in standalone lineage server ([feast-dev#6873](feast-dev#6873)) ([ecd0d38](feast-dev@ecd0d38)) * Keep a zero watermark delay threshold on a Kafka source ([e2e28c7](feast-dev@e2e28c7)) * Keep entity rows apart in the ibis point-in-time join ([a87c070](feast-dev@a87c070)) * Keep the BatchFeatureView type and settings when copying it ([feast-dev#6989](feast-dev#6989)) ([013fc57](feast-dev@013fc57)) * Keep tiling, offline and stream engine settings when copying a StreamFeatureView ([fac0f88](feast-dev@fac0f88)) * Limit tiled max and min aggregations to the window ([2583c45](feast-dev@2583c45)) * Load Milvus collections once instead of on every query ([feast-dev#6882](feast-dev#6882)) ([f9e91fa](feast-dev@f9e91fa)) * Make vector length validation reachable, schema-driven and vectorized ([feast-dev#6909](feast-dev#6909)) ([29af8c0](feast-dev@29af8c0)), closes [feast-dev#6907](feast-dev#6907) * Map INT32_LIST, PDF_BYTES and IMAGE_BYTES type names to their ValueType ([8c1e825](feast-dev@8c1e825)) * Map narrow and unsigned Arrow integer types to Feast value types ([feast-dev#6960](feast-dev#6960)) ([ae74360](feast-dev@ae74360)) * Map Spark smallint and tinyint columns to INT32 ([9d42729](feast-dev@9d42729)) * Map SQL Server bigint to INT64 and float to DOUBLE ([6d15533](feast-dev@6d15533)) * Map Trino DECIMAL columns to pyarrow decimal128 ([155bc24](feast-dev@155bc24)) * Normalize Arrow-backed dtypes from Ray Dataset.to_pandas() for pyarrow >=25 compatibility ([feast-dev#6916](feast-dev#6916)) ([374f1d7](feast-dev@374f1d7)) * **operator:** Set grpc appProtocol on the offline Service port ([f9dcfb5](feast-dev@f9dcfb5)) * Pin expected tensor to same device in online retrieval test ([feast-dev#6777](feast-dev#6777)) ([1f2584d](feast-dev@1f2584d)) * Preserve an unset aggregation time window across a proto round-trip ([2000379](feast-dev@2000379)) * Preserve entity row order in precomputed reads ([ce37029](feast-dev@ce37029)) * Pull the MinIO test image from pgsty/minio ([fe27230](feast-dev@fe27230)) * Pull the MinIO test image from quay ([feast-dev#6838](feast-dev#6838)) ([367001b](feast-dev@367001b)) * Raise Trino monitoring query errors other than a missing table ([6161766](feast-dev@6161766)) * Read Kafka and Kinesis sources without a batch source from proto ([a620f5c](feast-dev@a620f5c)) * Read naive datetimes as UTC in UnixTimestamp values ([bc703d9](feast-dev@bc703d9)) * Record which storage class wrote a custom_storage payload ([feast-dev#6988](feast-dev#6988)) ([c0545f7](feast-dev@c0545f7)) * Refresh the SQL registry cache after deleting a data source ([66c55e3](feast-dev@66c55e3)) * Refresh the SQL registry cache after deleting a permission ([6b212f5](feast-dev@6b212f5)) * Reject unknown resource types and actions in the REST permission API ([a5eaa36](feast-dev@a5eaa36)) * Remove pip install command from Redis GLIDE docstring ([66ff990](feast-dev@66ff990)) * Remove runtime pip install hints from mlflow errors ([dd9f099](feast-dev@dd9f099)) * Remove unsupported vector_len from PostgreSQL tutorial config ([fcc9a85](feast-dev@fcc9a85)) * Repair Chronon operator validation and Pixi CI setup ([feast-dev#6947](feast-dev#6947)) ([3710f78](feast-dev@3710f78)) * Report Dask monitoring read errors instead of no data ([81673d9](feast-dev@81673d9)) * Resolve feature view batch source name against registry ([f8883d0](feast-dev@f8883d0)) * Restrict UI server CORS to trusted origins ([feast-dev#6986](feast-dev#6986)) ([5e7ef3e](feast-dev@5e7ef3e)) * Return feature_view_metadata from the REST feature server ([feast-dev#6927](feast-dev#6927)) ([bc5aeef](feast-dev@bc5aeef)) * Serve aliased on demand feature views in feature services ([823b92e](feast-dev@823b92e)) * Snapshot tag filters in registry cache keys ([81e1546](feast-dev@81e1546)) * Stabilize materialization checks across hour boundaries ([feast-dev#6948](feast-dev#6948)) ([acadb17](feast-dev@acadb17)) * Stop the Couchbase online store reporting failed writes as written ([b3f3c98](feast-dev@b3f3c98)) * Support SQL registry dump ([b60f9c7](feast-dev@b60f9c7)) * Treat datetimes with a None utcoffset as naive ([687bee8](feast-dev@687bee8)) * Unwrap SecretStr tokens when building Trino auth objects ([feast-dev#6771](feast-dev#6771)) ([922db5c](feast-dev@922db5c)) * Use a valid ShuffleStrategy for the Ray DatasetContext ([6259f5c](feast-dev@6259f5c)) * Use do_exchange for offline server reads to support HPA ([bae07fc](feast-dev@bae07fc)) * Use group_by in Polars backend aggregation ([feast-dev#6965](feast-dev#6965)) ([7b9fad2](feast-dev@7b9fad2)) * Use upstream kubectl image as feast-operator CronJob default ([d6cc1dc](feast-dev@d6cc1dc)) * Use UTC end bound and correct docs for disable_event_timestamp ([feast-dev#6956](feast-dev#6956)) ([cc07c82](feast-dev@cc07c82)), closes [feast-dev#6936](feast-dev#6936) ### Features * Add a non-JVM read path for Lance data sources ([feast-dev#6943](feast-dev#6943)) ([e2a53bb](feast-dev@e2a53bb)), closes [feast-dev#6925](feast-dev#6925) [feast-dev#5782](feast-dev#5782) [feast-dev#6909](feast-dev#6909) * Add a non-JVM write path for Lance data sources ([feast-dev#6945](feast-dev#6945)) ([f754323](feast-dev@f754323)), closes [feast-dev#6943](feast-dev#6943) [feast-dev#6909](feast-dev#6909) * Add Chronon online and offline store integrations ([feast-dev#6188](feast-dev#6188)) ([dd059c6](feast-dev@dd059c6)) * Add consistency_level to the Milvus online store config ([97296c1](feast-dev@97296c1)) * Add dualStack server option to the Feast operator ([feast-dev#6887](feast-dev#6887)) ([725a147](feast-dev@725a147)), closes [feast-dev#6977](feast-dev#6977) * Add Lance optional dependencies ([feast-dev#6958](feast-dev#6958)) ([20bc993](feast-dev@20bc993)) * Add LanceFormat table format with version/tag pinning ([1196e22](feast-dev@1196e22)), closes [feast-dev#5650](feast-dev#5650) * Add MLflow as a first-class offline DataSource for Feast ([feast-dev#6702](feast-dev#6702)) ([3667772](feast-dev@3667772)) * Add optional GLIDE client for Redis online reads ([feast-dev#6870](feast-dev#6870)) ([6e9a532](feast-dev@6e9a532)) * Add quality monitoring features for Trino offline store ([feast-dev#6779](feast-dev#6779)) ([9340545](feast-dev@9340545)) * Add SavedDatasetLanceStorage ([1118d3e](feast-dev@1118d3e)) * Add standalone MCP server as a first-class component ([0efab5c](feast-dev@0efab5c)), closes [python-keycloak#623](https://github.com/python-keycloak/issues/623) * Add structured audit logging for MCP feature server ([feast-dev#6456](feast-dev#6456)) ([640ac70](feast-dev@640ac70)) * Add Trino compute engine for batch retrieval and materialization ([feast-dev#6898](feast-dev#6898)) ([11a47b4](feast-dev@11a47b4)) * Batch Postgres online reads across feature views ([feast-dev#6832](feast-dev#6832)) ([514bee7](feast-dev@514bee7)), closes [feast-dev#3259](feast-dev#3259) * **bigtable:** Make write batch size and concurrency configurable ([feast-dev#6847](feast-dev#6847)) ([5388522](feast-dev@5388522)) * Bucket feature count labels in latency metrics ([feast-dev#6811](feast-dev#6811)) ([d9ea120](feast-dev@d9ea120)) * Expose runtime version in Feast UI ([feast-dev#6843](feast-dev#6843)) ([fa8f06b](feast-dev@fa8f06b)) * Make Milvus index and search params configurable ([ef743ff](feast-dev@ef743ff)) * Offline write batch for IcebergSource ([5be5ed0](feast-dev@5be5ed0)) * OpenLineage consumer improvements - dataset versioning, column lineage, ownership ([69f294e](feast-dev@69f294e)) * **operator:** Add tolerations and nodeSelector to the FeatureStore CR ([feast-dev#6741](feast-dev#6741)) ([03dac6f](feast-dev@03dac6f)) * Serve the MCP server dual-stack on --host :: ([4fc9726](feast-dev@4fc9726)) * Support Milvus partition keys in the Milvus online store ([0ed3285](feast-dev@0ed3285)) * Support token, uri and db_name in the Milvus online store ([c5efecd](feast-dev@c5efecd)) * Version-pinning for FeatureService (online + offline) ([feast-dev#6718](feast-dev#6718)) ([787845a](feast-dev@787845a)) ### Performance Improvements * Avoid pandas copy in Spark reads ([feast-dev#6878](feast-dev#6878)) ([4771156](feast-dev@4771156)) * Decouple metrics HTTP server into a dedicated child process ([feast-dev#6855](feast-dev#6855)) ([0b621de](feast-dev@0b621de)) * Reduce online serving hot-path overhead from metrics and audit logging ([5ee7ee3](feast-dev@5ee7ee3)) --------- Signed-off-by: Srihari <svenkata@redhat.com> Signed-off-by: ntkathole <nikhilkathole2683@gmail.com> Co-authored-by: Srihari <svenkata@redhat.com> Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: feast-ci-bot <feast-ci-bot@willem.co>

haoxu0 requested a review from a team as a code owner October 5, 2026 20:31

haoxu0 mentioned this pull request Oct 5, 2026

SavedDatasetStorage registry: custom_storage is claimed by three classes, so from_proto dispatch depends on import order #6946

Closed

haoxu0 added the ok-to-test label Oct 5, 2026

haoxu0 requested review from franciscojavierarceo, ntkathole and shuchu October 6, 2026 06:54

haoxu0 mentioned this pull request Oct 7, 2026

feat: Add Lance optional dependencies #6958

Merged

haoxu0 force-pushed the feat/lance-write branch from f30b007 to e57bbb8 Compare October 7, 2026 06:06

ntkathole reviewed Oct 9, 2026

View reviewed changes

ntkathole approved these changes Oct 9, 2026

View reviewed changes

haoxu0 added 3 commits October 9, 2026 08:49

ntkathole force-pushed the feat/lance-write branch from 55b76e1 to ee23843 Compare October 9, 2026 03:19

ntkathole merged commit f754323 into feast-dev:master Oct 9, 2026
20 of 23 checks passed

rhods-devops-app Bot mentioned this pull request Oct 14, 2026

Automated Sync from main to stable red-hat-data-services/feast#493

Merged

Navigation Menu

Repository navigation

feat: Add a non-JVM write path for Lance data sources - #6945

feat: Add a non-JVM write path for Lance data sources#6945
ntkathole merged 3 commits into
feast-dev:masterfrom
haoxu0:feat/lance-write

haoxu0 commented Oct 5, 2026 •

edited

Loading

Uh oh!

codecov-commenter commented Oct 5, 2026 •

edited

Loading

Uh oh!

haoxu0 commented Oct 9, 2026

Uh oh!

ntkathole Oct 9, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

haoxu0 commented Oct 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What this does

How far the namespace is involved

Three things settled before Lance is called

Declared vector_length

Mode semantics

Verification

Uh oh!

codecov-commenter commented Oct 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

haoxu0 commented Oct 9, 2026

Uh oh!

ntkathole Oct 9, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

haoxu0 commented Oct 5, 2026 •

edited

Loading

codecov-commenter commented Oct 5, 2026 •

edited

Loading