Repository navigation
Cant retrieve the entity field #5156
Description
Expected Behavior
When retrieving documents from an online store we don't care about the embedding after the top k retrieval we only care about the document ID, or some other metadata.
The current behavior doesn't allow retrieval of any field other than the "embedding" field, which is problematic.
Current Behavior
Can't retrieve the field other then the
dtype=Array(Float32),
vector_index=True,
field.
Steps to reproduce
Create a feature view with 2 fields:
item_embedding_view = FeatureView(
name="item_embedding",
entities=["item_id"],
ttl=timedelta(days=365 * 5),
schema=[
Field(name="item_id", dtype=Int64),
Field(
name="embedding",
dtype=Array(Float32),
vector_index=True,
vector_search_metric="cosine",
),
],
source=item_embed_push_source,
online=True
)
And push a DataFrame to the online store
item_embed_df:
item_id embedding event_timestamp
0 167 [0.12210895866155624, -0.11489687114953995, -1... 2025-03-17 09:53:12.138190
1 1003 [-0.9383772015571594, -1.7450543642044067, 0.7... 2025-03-17 09:53:12.138190
2 4895 [-0.9383772015571594, -1.7450543642044067, 0.7... 2025-03-17 09:53:12.138190
3 3812 [-0.8219010829925537, 0.1411619633436203, 1.12... 2025-03-17 09:53:12.138190
4 4941 [-0.6469292044639587, -1.1105291843414307, -0.... 2025-03-17 09:53:12.138190
... ... ... ...
4995 1216 [0.12210895866155624, -0.11489687114953995, -1... 2025-03-17 09:53:12.138190
4996 4198 [-0.6469292044639587, -1.1105291843414307, -0.... 2025-03-17 09:53:12.138190
4997 112 [0.5381854176521301, 0.2744300961494446, -1.46... 2025-03-17 09:53:12.138190
4998 2948 [-0.8219010829925537, 0.1411619633436203, 1.12... 2025-03-17 09:53:12.138190
4999 4708 [-0.8219010829925537, 0.1411619633436203, 1.12... 2025-03-17 09:53:12.138190
Push and materialize:
store.push('item_embed_push_source', item_embed_df, to=PushMode.ONLINE)
store.materialize_incremental(datetime.now(), feature_views=['item_embedding'])
Try to retrieve item_embedding:item_id' when using the code:
store.retrieve_online_documents(
query=user_embed,
top_k=64,
feature='item_embedding:item_id'
).to_df()
Returns an empty Dataframe item_id distance when trying to retrieve entity field, and returns an empty list when trying to retrieve other fields (assume I added it to the FV)
other_field distance
0 [] None
1 [] None
... ... ...
62 [] None
63 [] None
NOTE: If I call the feature 'item_embedding:embedding' instead of 'item_embedding:item_id, I will get results.
Specifications
- Version: '0.46.0'
- Platform: PGvector locally
- Subsystem: Fedora
Possible Solution
Add the options to do it
Activity
franciscojavierarceo commentedon Mar 17, 2025
@ntkathole @jyejare do either of you two have bandwidth to investigate?
jyejare commentedon Mar 19, 2025
Yes, we can reproduce the issue here:
Will look into this further.
jyejare commentedon Mar 27, 2025
@itay1551 The issue is now fixed, and overal, thel document retrieval experience is changed.