Repository navigation
Online store latency (redis) is growing over time #3597
Description
Expected Behavior
The online-store latency remains the same and doesn't grow over the time. We assume that amount of data in redis is not growing.
Current Behavior
The median latency growth over the past 30 days is on the chart above.
Steps to reproduce
We use a custom aiohttp python service to relay requests to the online-store, where we invoke the native feast client. The service is completely stateless, and I don't expect it to be the source of the problem.
here is our feast config:
config = RepoConfig(
project='feast',
registry='gs://our-bucket/feast/registry_file_3.db',
provider='gcp',
online_store=RedisOnlineStoreConfig(
connection_string=get_redis_connection_string(),
redis_type='redis',
key_ttl_seconds=REDIS_TTL_SECONDS,
),
offline_store=BigQueryOfflineStoreConfig(
dataset='feature_store',
project_id='project-id',
),
entity_key_serialization_version=2,
)
as you see, we use the default registry cache TTL (=600).
Specifications
- Version: 0.28.0
- Platform: amd64
- Subsystem: debian 10
Possible Solution
We noticed that changing the path to file-based registry (i.e. effectively re-creating it) eliminates the latency growth and it back to normal (today's chart):
Therefore, a solution might be related to fixing the registry caching mechanism.
Let me know if further details are needed!
Activity
RadionBik commentedon Apr 14, 2023
I have noticed that timestamps of incremental materialization are stored in the registry and are sent to client as well. We run incr. mat. every 15 min, so over month it yields to 30 * 24 * 4 = 2880 timestamps per view, which might explain the gradual increase of the response time. Not sure if it is the reason, but decided to share the info here.
adchia commentedon Apr 21, 2023
Thanks for filing! do you mean that you changed from the file based registry to have a TTL=0 and it was ok?
RadionBik commentedon Apr 21, 2023
I have never adjusted registry cache TTL, it has always been set to the default value.
nturusin commentedon May 5, 2023
Hi guys. Any news on this? This issue forces us to update the registry file in production every couple of weeks to reset the latency..
nturusin commentedon May 23, 2023
Hi @adchia
Do you have any plans to fix this soon?
RadionBik commentedon May 30, 2023
an update from us:
having disabled incremental materialization for a week, we do NOT notice increases in latency anymore, which confirms my hypothesis above. As you at the chart below, we recreated the registry and disabled incr. mat. around 25th of May:
Unfortunately, the cause nor a resolution of it is not obvious for me ATM.
stale commentedon Oct 15, 2023
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
RadionBik commentedon Nov 8, 2023
we found that logging to feast caused the problem, we disabled it and everything runs without leaks now
tokoko commentedon Mar 23, 2024
@RadionBik hey, thanks for investigating this, can you clarify the last comments? Did you find that both incremental materialization payload and
feast usagewere the culprits here?RadionBik commentedon Mar 23, 2024
I might have left the last comment in a wrong issue.
So the current status is:
Hope this clarifies the situation a bit
tokoko commentedon Mar 28, 2024
@RadionBik thanks for the clarification
stale commentedon Apr 26, 2025
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.