Eliminate vector length restrictions in the store

Expected Behavior

Let's say I have two arrays in a store, the first one of size k and the other one of size m
In the current behavior of Feast, we define in the feature_store.yaml

online_store:
  ...
  vector_len: 64
  ...

Here k == 64 is the size of the embedding vector that will be used in the online store,
If I want to push to the store an array that isn't of size k == 64 I will get an error even if the array is defined as vector_index=False in the Feature View,

From my perspective, the vector_len parameter needs to be defined per field in a FeatureView, or even not predefined.

Current Behavior

When pushing to a store a vector of size m when m != k where in the feature_repo.yaml defined vector_len: k
an error rise
DataException: expected 64 dimensions, not 10

Steps to reproduce

Create feature view and push a vector of size that is different from the defined feature_repo.yaml file

Specifications

Version: 0.46.0
Platform: local
Subsystem: Fedora

Possible Solution

This is a good first issue. The vector_len is now able to be defined at the Field level like so:

document_embeddings = FeatureView(
    name="embedded_documents",
    entities=[item, author],
    schema=[
        Field(
            name="vector",
            dtype=Array(Float32),
            # Look how easy it is to enable RAG!
            vector_index=True,
            vector_search_metric="COSINE",
        ),
        Field(name="item_id", dtype=Int64),
        Field(name="author_id", dtype=String),
        Field(name="created_timestamp", dtype=UnixTimestamp),
        Field(name="sentence_chunks", dtype=String),
        Field(name="event_timestamp", dtype=UnixTimestamp),
    ],
    source=rag_documents_source,
    ttl=timedelta(hours=24),
)

So we can remove this dependency.

Realizing now that this is not as straightforward.

Need to update the Field protobuf object to include vector length, sorry I misunderstood the ticket. I need some sleep 😭

/me taking this up.

Navigation Menu

Repository navigation

Eliminate vector length restrictions in the store #5166

Expected Behavior

Current Behavior

Steps to reproduce

Specifications

Possible Solution

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Eliminate vector length restrictions in the store #5166

Description

Expected Behavior

Current Behavior

Steps to reproduce

Specifications

Possible Solution

Activity

franciscojavierarceo commented on Apr 1, 2025

franciscojavierarceo commented on Apr 2, 2025

jyejare commented on Apr 9, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions