← 输入框|github.com|刷新
Skip to content

feat: Add native_numeric_types to the Milvus online store - #7025

Open
simonhearne wants to merge 3 commits into
feast-dev:masterfrom
simonhearne:feat/milvus-native-numeric-types
Open

simonhearne wants to merge 3 commits into
feast-dev:masterfrom
simonhearne:feat/milvus-native-numeric-types

Conversation

@simonhearne

Copy link
Copy Markdown
Contributor

What this PR does / why we need it:

The Milvus online store stores numeric and bool features as VARCHAR unless
enable_openai_compatible_store is set. Range filters (gt, gte, lt, lte) on those fields
compare strings, so "9" > "100" is true. The flag that fixes this is shared with the SQLite,
Postgres and Elasticsearch stores, and its name doesn't say that it controls numeric storage.

This PR adds native_numeric_types as a Milvus-specific name for the same behaviour. Either flag
stores numeric and bool features as native INT32, INT64, FLOAT, DOUBLE and BOOL fields.
The warning logged for numeric range filters on VARCHAR collections now points to the new flag.

online_store:
  type: milvus
  native_numeric_types: true

The docs describe the migration: set the flag, run feast teardown and feast apply, then
materialize again. Collections that haven't been recreated keep working, because Feast detects the
storage type of each collection.

If you would rather not add a Milvus-only alias for a cross-store flag, I can reduce this PR to
the docs and the updated warning.

Which issue(s) this PR fixes:

N/A

Checks

  • I've made sure the tests are passing.
  • My commits are signed off (git commit -s)
  • My PR title follows conventional commits format

Testing Strategy

  • Unit tests

  • Integration tests

  • Manual tests

  • Testing is not required for this change

  • Unit (Milvus Lite): field types with the default config, with native_numeric_types and with
    enable_openai_compatible_store; a gt filter compares numerically; values round-trip.

Unit tests run on Milvus Lite 3.2.1 (pymilvus 3.0.2). The existing server tests in sdk/python/tests/integration/online_store/test_milvus_remote.py pass against a local Milvus 2.6.0, a local Milvus 3.0.2 server and Zilliz Cloud. The existing Milvus unit and universal integration tests pass unchanged.

Misc

Part of a series of Milvus online store improvements for Zilliz Cloud and production Milvus, following #6882, #6895, #6917, #6921 and #6940.

Milvus stores numeric features as VARCHAR unless
enable_openai_compatible_store is set, so range filters compare
lexicographically. That flag is shared by several online stores, and
its name doesn't say that for Milvus it only switches numeric storage.

Adds native_numeric_types as a Milvus-specific, clearly named way to
store numeric and bool features as native Milvus types. Either flag
enables it, so existing configs are unaffected. The warning for
numeric filters on VARCHAR collections now points to the new flag, and
the docs describe how to migrate existing collections.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Simon Hearne <simon.hearne@gmail.com>
@simonhearne
simonhearne requested a review from a team as a code owner October 16, 2026 09:05
@codecov-commenter

codecov-commenter commented Oct 16, 2026 •

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 51.02%. Comparing base (e304a84) to head (206309f).
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #7025      +/-   ##
==========================================
+ Coverage   50.99%   51.02%   +0.03%     
==========================================
  Files         455      455              
  Lines       56994    56997       +3     
  Branches     8321     8321              
==========================================
+ Hits        29063    29082      +19     
+ Misses      25946    25934      -12     
+ Partials     1985     1981       -4     
Flag Coverage Δ
go-feature-server 30.58% <ø> (ø)
python-unit 52.48% <100.00%> (+0.03%) ⬆️
Files with missing lines Coverage Δ
.../infra/online_stores/milvus_online_store/milvus.py 79.41% <100.00%> (+3.24%) ⬆️

Continue to review full report in Codecov by Harness.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e304a84...206309f. Read the comment docs.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@simonhearne

Copy link
Copy Markdown
Contributor Author

@ntkathole LMK if you have any thoughts on this PR.

haoxu0 commented Oct 23, 2026

Copy link
Copy Markdown
Collaborator

The numeric filter correctness fix and the backward-compatible alias look reasonable. One migration concern: the documented feast teardown → feast apply → rematerialize sequence drops the existing collection before it is repopulated, so this can create a serving outage and potentially lose online-only state. Could we add an explicit destructive / downtime warning and recommend a backup or staged migration (e.g. a new collection with cutover) for production deployments? Also, the current tests establish the newly created native-typed collection, but could we add a regression that creates a VARCHAR collection first, flips native_numeric_types=True without teardown, then verifies reads/writes still work and numeric range filters continue to warn/use the existing storage semantics? This would pin down the promised mixed old/new collection behavior.

Teardown drops every collection in the project and loses online-only
rows, so the docs now warn about it and describe a staged cutover that
avoids it.

Numeric range filters on VARCHAR collections don't compare
lexicographically as the warning and docs said: Milvus rejects numeric
filter values on VARCHAR fields, and only string values compare as
text. Correct both, and add a regression test that turns on
native_numeric_types over an existing VARCHAR collection.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Simon Hearne <simon.hearne@gmail.com>
@simonhearne

Copy link
Copy Markdown
Contributor Author

@haoxu0 thank you for the review, both points are addressed in 206309f.
Let me know if there's anything else - could a maintainer rerun the failed unit-test-python check?

Migration warning. The Numeric types section now has a warning hint: feast teardown drops the collections of every feature view in the project, online reads fail until materialization finishes, and rows that only exist in the online store (push sources, write_to_online_store) are lost. The section also documents a staged cutover that needs no teardown: add a copy of the feature view under a new name (which creates a new native-typed collection), materialize it, move readers to it, then remove the old view. It points to Milvus Backup for anyone who still wants to tear down.

Mixed-collection regression. test_native_numeric_types_keeps_existing_varchar_collection creates a VARCHAR collection, turns on native_numeric_types without a teardown, and checks that the collection stays VARCHAR and that writes and online_read still round-trip.

Writing that test showed that the existing warning (and my docs) described the wrong behaviour on VARCHAR collections. A numeric filter value doesn't compare lexicographically: Milvus rejects it (comparisons between VarChar and Int64 are not supported on Milvus 3.0.2, with a similar error on Lite). Only a string value like "50" compares as text. The test now pins both cases and the warning, and I've corrected the warning text and docs to match. I left the filter behaviour itself unchanged: an error is safer than silently wrong results.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants