Repository navigation
feat: Add native_numeric_types to the Milvus online store - #7025
simonhearne wants to merge 3 commits into
Conversation
Milvus stores numeric features as VARCHAR unless enable_openai_compatible_store is set, so range filters compare lexicographically. That flag is shared by several online stores, and its name doesn't say that for Milvus it only switches numeric storage. Adds native_numeric_types as a Milvus-specific, clearly named way to store numeric and bool features as native Milvus types. Either flag enables it, so existing configs are unaffected. The warning for numeric filters on VARCHAR collections now points to the new flag, and the docs describe how to migrate existing collections. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Simon Hearne <simon.hearne@gmail.com>
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #7025 +/- ##
==========================================
+ Coverage 50.99% 51.02% +0.03%
==========================================
Files 455 455
Lines 56994 56997 +3
Branches 8321 8321
==========================================
+ Hits 29063 29082 +19
+ Misses 25946 25934 -12
+ Partials 1985 1981 -4
Continue to review full report in Codecov by Harness.
🚀 New features to boost your workflow:
|
|
@ntkathole LMK if you have any thoughts on this PR. |
|
The numeric filter correctness fix and the backward-compatible alias look reasonable. One migration concern: the documented |
Teardown drops every collection in the project and loses online-only rows, so the docs now warn about it and describe a staged cutover that avoids it. Numeric range filters on VARCHAR collections don't compare lexicographically as the warning and docs said: Milvus rejects numeric filter values on VARCHAR fields, and only string values compare as text. Correct both, and add a regression test that turns on native_numeric_types over an existing VARCHAR collection. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Simon Hearne <simon.hearne@gmail.com>
|
@haoxu0 thank you for the review, both points are addressed in 206309f. Migration warning. The Numeric types section now has a warning hint: Mixed-collection regression. Writing that test showed that the existing warning (and my docs) described the wrong behaviour on VARCHAR collections. A numeric filter value doesn't compare lexicographically: Milvus rejects it ( |
What this PR does / why we need it:
The Milvus online store stores numeric and bool features as VARCHAR unless
enable_openai_compatible_storeis set. Range filters (gt,gte,lt,lte) on those fieldscompare strings, so
"9" > "100"is true. The flag that fixes this is shared with the SQLite,Postgres and Elasticsearch stores, and its name doesn't say that it controls numeric storage.
This PR adds
native_numeric_typesas a Milvus-specific name for the same behaviour. Either flagstores numeric and bool features as native
INT32,INT64,FLOAT,DOUBLEandBOOLfields.The warning logged for numeric range filters on VARCHAR collections now points to the new flag.
The docs describe the migration: set the flag, run
feast teardownandfeast apply, thenmaterialize again. Collections that haven't been recreated keep working, because Feast detects the
storage type of each collection.
If you would rather not add a Milvus-only alias for a cross-store flag, I can reduce this PR to
the docs and the updated warning.
Which issue(s) this PR fixes:
N/A
Checks
git commit -s)Testing Strategy
Unit tests
Integration tests
Manual tests
Testing is not required for this change
Unit (Milvus Lite): field types with the default config, with
native_numeric_typesand withenable_openai_compatible_store; agtfilter compares numerically; values round-trip.Unit tests run on Milvus Lite 3.2.1 (pymilvus 3.0.2). The existing server tests in
sdk/python/tests/integration/online_store/test_milvus_remote.pypass against a local Milvus 2.6.0, a local Milvus 3.0.2 server and Zilliz Cloud. The existing Milvus unit and universal integration tests pass unchanged.Misc
Part of a series of Milvus online store improvements for Zilliz Cloud and production Milvus, following #6882, #6895, #6917, #6921 and #6940.