← 输入框|github.com|刷新
Skip to content

feat: Add SavedDatasetLanceStorage - #7001

Merged
ntkathole merged 1 commit into
feast-dev:masterfrom
haoxu0:feat/saved-dataset-lance-storage
Oct 13, 2026
Merged

ntkathole merged 1 commit into
feast-dev:masterfrom
haoxu0:feat/saved-dataset-lance-storage

Conversation

@haoxu0

@haoxu0 haoxu0 commented Oct 13, 2026

Copy link
Copy Markdown
Collaborator

What this PR does / why we need it:

Adds SavedDatasetLanceStorage, so an existing Lance dataset can be recorded as
a saved dataset without moving or rewriting it.

This is the path for a dataset produced outside Feast. create_saved_dataset
takes a RetrievalJob, executes it and persists the result, which is wrong for
an embedding dataset that a Ray or Spark job already wrote to Lance.
BaseRegistry.apply_saved_dataset records metadata only, and that is what this
storage class is for:

storage = SavedDatasetLanceStorage(LanceFormat(version=7), uri="s3://bucket/embeddings")
registry.apply_saved_dataset(
    SavedDataset(
        name="poi_embeddings_v7",
        features=[],
        join_keys=[],
        storage=storage,
        tags={"model": "clip-vit-l-14", "recall_at_10": "0.9137"},
    ),
    project,
)

The format must pin a version

SavedDatasetLanceStorage rejects an unpinned LanceFormat. A saved dataset
addressing whatever version is latest at read time does not identify the rows it
was created from, so a measurement recorded against it stops being attributable.
version is preferable to tag, since a tag can be moved afterwards.

Note that LanceSource defaults table_format to an unpinned LanceFormat
rather than None, so from_data_source on a plain source raises rather than
silently recording a drifting reference.

Dispatch

Lance is the fourth class to claim the custom_storage oneof field, after
postgres, clickhouse and couchbase. to_proto therefore goes through
SavedDatasetStorage._custom_storage_proto (#6988) so the payload records the
writing class; otherwise from_proto would resolve by import order.

Verified that a Lance payload reads back as SavedDatasetLanceStorage with all
four claimants registered, across several import orders:

[postgres clickhouse couchbase lance]  claimants=4  -> SavedDatasetLanceStorage  version=7
[lance postgres clickhouse couchbase]  claimants=4  -> SavedDatasetLanceStorage  version=7
[clickhouse lance couchbase postgres]  claimants=4  -> SavedDatasetLanceStorage  version=7

LanceSource is also registered in _DATA_SOURCE_TO_SAVED_DATASET_STORAGE, so
SavedDatasetStorage.from_data_source resolves it.

Which issue(s) this PR fixes:

N/A. Follows #6988, which made the shared custom_storage field safe for a
fourth claimant.

Checks

  • I've made sure the tests are passing.
  • My commits are signed off (git commit -s)
  • My PR title follows conventional commits format

Testing Strategy

  • Unit tests
  • Integration tests
  • Manual tests
  • Testing is not required for this change

sdk/python/tests/unit/infra/data_sources/test_saved_dataset_lance_storage.py,
14 tests: round trip by uri and by namespace+table, the payload recording the
storage class, an unpinned format refused, a wrong-typed format refused, uri and
table exclusivity, a table without namespace_impl refused, to_data_source
and from_data_source preserving the pin, a non-Lance source refused, the
_DATA_SOURCE_TO_SAVED_DATASET_STORAGE registration, and dispatch with all four
custom_storage claimants present.

Against master: ruff check and ruff format --check clean, and mypy feast
reports the same 24 pre-existing errors as origin/master, none added. The
adjacent merged suites (tests/unit/infra/data_sources/,
tests/unit/test_table_format.py, tests/unit/test_saved_dataset_storage_dispatch.py)
show an identical set of failures to clean master, all in
test_mlflow_dataset_source.py from a missing local dependency.

@haoxu0
haoxu0 requested a review from a team as a code owner October 13, 2026 07:09
@codecov-commenter

codecov-commenter commented Oct 13, 2026 •

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 50.92%. Comparing base (013fc57) to head (e72dd18).
⚠️ Report is 2 commits behind head on master.
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #7001      +/-   ##
==========================================
+ Coverage   50.89%   50.92%   +0.02%     
==========================================
  Files         452      452              
  Lines       56621    56654      +33     
  Branches     8251     8256       +5     
==========================================
+ Hits        28819    28852      +33     
  Misses      25821    25821              
  Partials     1981     1981              
Flag Coverage Δ
go-feature-server 30.58% <ø> (ø)
python-unit 52.39% <100.00%> (+0.02%) ⬆️
Files with missing lines Coverage Δ
...t/infra/data_sources/contrib/lance/lance_source.py 92.23% <100.00%> (+1.37%) ⬆️
sdk/python/feast/saved_dataset.py 65.86% <ø> (ø)

Continue to review full report in Codecov by Harness.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 013fc57...e72dd18. Read the comment docs.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Records an existing Lance dataset as a saved dataset, without moving or
rewriting it. This is the path for a dataset produced outside Feast, by a Ray or
Spark job: FeatureStore.create_saved_dataset cannot serve it, because it
executes a RetrievalJob and persists the result, while
BaseRegistry.apply_saved_dataset records metadata only.

The format must pin a version or a tag. A saved dataset addressing the latest
version does not identify the rows it was created from, so a measurement
recorded against it stops being attributable.

Lance is the fourth class to claim the custom_storage oneof field, so to_proto
goes through SavedDatasetStorage._custom_storage_proto and the payload records
the writing class. Verified that a Lance payload reads back as
SavedDatasetLanceStorage with postgres, clickhouse and couchbase also imported,
in several import orders.

LanceSource is registered in _DATA_SOURCE_TO_SAVED_DATASET_STORAGE so
SavedDatasetStorage.from_data_source resolves it.

Signed-off-by: hao-xu5 <hxu44@apple.com>
@ntkathole
ntkathole force-pushed the feat/saved-dataset-lance-storage branch from 0eed6e8 to e72dd18 Compare October 13, 2026 09:42
@ntkathole
ntkathole merged commit 1118d3e into feast-dev:master Oct 13, 2026
22 of 23 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants