Repository navigation
feat: Add SavedDatasetLanceStorage - #7001
Merged
ntkathole merged 1 commit intoOct 13, 2026
Merged
Conversation
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #7001 +/- ##
==========================================
+ Coverage 50.89% 50.92% +0.02%
==========================================
Files 452 452
Lines 56621 56654 +33
Branches 8251 8256 +5
==========================================
+ Hits 28819 28852 +33
Misses 25821 25821
Partials 1981 1981
Continue to review full report in Codecov by Harness.
🚀 New features to boost your workflow:
|
ntkathole
approved these changes
Oct 13, 2026
Records an existing Lance dataset as a saved dataset, without moving or rewriting it. This is the path for a dataset produced outside Feast, by a Ray or Spark job: FeatureStore.create_saved_dataset cannot serve it, because it executes a RetrievalJob and persists the result, while BaseRegistry.apply_saved_dataset records metadata only. The format must pin a version or a tag. A saved dataset addressing the latest version does not identify the rows it was created from, so a measurement recorded against it stops being attributable. Lance is the fourth class to claim the custom_storage oneof field, so to_proto goes through SavedDatasetStorage._custom_storage_proto and the payload records the writing class. Verified that a Lance payload reads back as SavedDatasetLanceStorage with postgres, clickhouse and couchbase also imported, in several import orders. LanceSource is registered in _DATA_SOURCE_TO_SAVED_DATASET_STORAGE so SavedDatasetStorage.from_data_source resolves it. Signed-off-by: hao-xu5 <hxu44@apple.com>
ntkathole
force-pushed
the
feat/saved-dataset-lance-storage
branch
from
October 13, 2026 09:42
0eed6e8 to
e72dd18
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this PR does / why we need it:
Adds
SavedDatasetLanceStorage, so an existing Lance dataset can be recorded asa saved dataset without moving or rewriting it.
This is the path for a dataset produced outside Feast.
create_saved_datasettakes a
RetrievalJob, executes it and persists the result, which is wrong foran embedding dataset that a Ray or Spark job already wrote to Lance.
BaseRegistry.apply_saved_datasetrecords metadata only, and that is what thisstorage class is for:
The format must pin a version
SavedDatasetLanceStoragerejects an unpinnedLanceFormat. A saved datasetaddressing whatever version is latest at read time does not identify the rows it
was created from, so a measurement recorded against it stops being attributable.
versionis preferable totag, since a tag can be moved afterwards.Note that
LanceSourcedefaultstable_formatto an unpinnedLanceFormatrather than
None, sofrom_data_sourceon a plain source raises rather thansilently recording a drifting reference.
Dispatch
Lance is the fourth class to claim the
custom_storageoneof field, afterpostgres, clickhouse and couchbase.
to_prototherefore goes throughSavedDatasetStorage._custom_storage_proto(#6988) so the payload records thewriting class; otherwise
from_protowould resolve by import order.Verified that a Lance payload reads back as
SavedDatasetLanceStoragewith allfour claimants registered, across several import orders:
LanceSourceis also registered in_DATA_SOURCE_TO_SAVED_DATASET_STORAGE, soSavedDatasetStorage.from_data_sourceresolves it.Which issue(s) this PR fixes:
N/A. Follows #6988, which made the shared
custom_storagefield safe for afourth claimant.
Checks
Testing Strategy
sdk/python/tests/unit/infra/data_sources/test_saved_dataset_lance_storage.py,14 tests: round trip by uri and by namespace+table, the payload recording the
storage class, an unpinned format refused, a wrong-typed format refused, uri and
table exclusivity, a table without
namespace_implrefused,to_data_sourceand
from_data_sourcepreserving the pin, a non-Lance source refused, the_DATA_SOURCE_TO_SAVED_DATASET_STORAGEregistration, and dispatch with all fourcustom_storageclaimants present.Against master:
ruff checkandruff format --checkclean, andmypy feastreports the same 24 pre-existing errors as
origin/master, none added. Theadjacent merged suites (
tests/unit/infra/data_sources/,tests/unit/test_table_format.py,tests/unit/test_saved_dataset_storage_dispatch.py)show an identical set of failures to clean master, all in
test_mlflow_dataset_source.pyfrom a missing local dependency.