performance issues of getting online features related to parsing protobuf data #3649

Activity

1. heavenmarshal

   added

   kind/featureNew feature or requestNew feature or request

   on Jun 12, 2023
2. goodseog commented on Jun 14, 2023

   

   goodseog

   on Jun 14, 2023

   Contributor

   More actions

   I'm facing the same issue too.

   From here, get_feature_views_to_use doesn't seem like a good idea to cache this because the feature_views to use changes depending on input argument features.

   feast/sdk/python/feast/feature_store.py

   Lines 2149 to 2154 in c474ccd

   	def _get_feature_views_to_use(
   	self,
   	features: Optional[Union[List[str], FeatureService]],
   	allow_cache=False,
   	hide_dummy_entity: bool = True,
   	) -> Tuple[List[FeatureView], List[RequestFeatureView], List[OnDemandFeatureView]]:

   What I found is, if feature registry has not expired, I assume that the registry.list_* methods always return the same result. Currently, this isn't cached, so list_* methods are being called every time, and from_proto in them is being called repeatedly and it makes get_online_features slow.
   (list_* methods mean registry.list_feature_views, registry.list_data_sources ...)

   So, my idea is to process the list_* methods using cached results when the registry is not expired.
   

   feast/sdk/python/feast/infra/registry/sql.py

   Lines 244 to 259 in c474ccd

   	def _refresh_cached_registry_if_necessary(self):
   	with self._refresh_lock:
   	expired = (
   	self.cached_registry_proto is None
   	or self.cached_registry_proto_created is None
   	) or (
   	self.cached_registry_proto_ttl.total_seconds()
   	> 0 # 0 ttl means infinity
   	and (
   	datetime.utcnow()
   	> (
   	self.cached_registry_proto_created
   	+ self.cached_registry_proto_ttl
   	)
   	)
   	)

   
   When using the sql registry, as you can see in the code above, the registry is updated by checking whether it has expired. Using this expired will tell us whether to return cached results. And I believe that caching this registry in memory would be no problem.
3. goodseog commented on Jun 14, 2023

   

   goodseog

   on Jun 14, 2023

   Contributor

   More actions

   or caching these fvs, request_fvs, od_fvs!

   feast/sdk/python/feast/feature_store.py

   Lines 2149 to 2177 in c474ccd

   	def _get_feature_views_to_use(
   	self,
   	features: Optional[Union[List[str], FeatureService]],
   	allow_cache=False,
   	hide_dummy_entity: bool = True,
   	) -> Tuple[List[FeatureView], List[RequestFeatureView], List[OnDemandFeatureView]]:
   	fvs = {
   	fv.name: fv
   	for fv in [
   	*self._list_feature_views(allow_cache, hide_dummy_entity),
   	*self._registry.list_stream_feature_views(
   	project=self.project, allow_cache=allow_cache
   	),
   	]
   	}
   	request_fvs = {
   	fv.name: fv
   	for fv in self._registry.list_request_feature_views(
   	project=self.project, allow_cache=allow_cache
   	)
   	}
   	od_fvs = {
   	fv.name: fv
   	for fv in self._registry.list_on_demand_feature_views(
   	project=self.project, allow_cache=allow_cache
   	)
   	}
4. felixwang9817

   closed this as completedin #3702on Sep 12, 2023