Add AWS native support for Feast #367
Description
Is your feature request related to a problem? Please describe.
Data Ingestion, feature query and some other components relies on GCP services like BidQuery, DataFlow. In order to make feast work on AWS. We'd like to add support for AWS using services like Athena, Glue or EMR.
Currently, AWS doesn't have managed Apache Beam jobs support, but there's lots of alternatives like Beam with Spark or Flink runner.
Describe the solution you'd like
- We need to make interface generic enough to support different cloud providers.
- Add AWS services support in feast.
Describe alternatives you've considered
Additional context
We'd like to know if feast internally already have some discussion on this. I am from AWS and would love to take design and implementation work.
Reacted by Mihail Petkov, Joel Cooklin, Theofilos Papapanagiotou, sbrooks-aither, Trevor Bortins, Gautam Kumar, dr3s, David Przybilla, Wei Qiu, Krzysztof Zarzycki and 11 more
Activity
woop commentedon Dec 14, 2019
Hi @Jeffwan,
Thank you for the interest. We definitely want to support non-GCP products. We already have @ches @smadarasmi who are planning to run Feast on-prem, which means they are implementing different store and runner types.
Other than that we've had a lot of demand for Feast to have either AWS or open source store/runner support. It's probably the most asked for functionality.
I agree with your proposal. We absolutely have to have generic interfaces that are easy to extend to various providers/technologies. We are currently close to cutting 0.4 which brings one or two important changes (project namespacing, async job management). That should land in 2 weeks.
Directly after that I would love to collaborate on an RFC for adding non-GCP technologies.
Which AWS store would you say is the higher priority to implement first? Redshift or Athena?
sbrooks-cerity commentedon Feb 13, 2020
Hi @woop
The Data Science team I work with has a huge interest in Feast but we are an AWS only shop. After discussions with them they don't have any hard requirements that necessitate either Athena or Redshift so for us our vote would be on whichever route would be the least path of resistance for integration and we can adjust fire on our end to leverage Feast.
Jeffwan commentedon Feb 13, 2020
@sbrooks-cerity do you have any specific requirements on where to run beam jobs?
Jeffwan commentedon Feb 13, 2020
@woop Sorry getting so long to come back to the issue. We have some internal discussion and plan to put efforts to add it. Athena will get higher priority. The challenge we have now is we don't have managed beam service. Seems either kinesis analytics, EMR or EKS (with operator runner) can support beam application. We'd like to hear more feedbacks from community users and make the decision.
woop commentedon Feb 15, 2020
Thanks for staying involved @Jeffwan!
I wonder if we should be focusing on supporting Flink or Spark (#362) instead of a managed Beam service (like Dataflow). This would also make it easier for us to support runners in our dev/testing environment.
I am also open to adding Athena support, but our focus in 0.6 will probably be to first add support for an on-prem warehouse. I will create a an issue to enumerate all of the Feast extension points. Hopefully this can become the basis for discussions of how Feast can be extended and the functionality that these components must provide.
Jeffwan commentedon Feb 18, 2020
Definitely. Managed service is ok, we need to build abstraction and make it supports other runners as well.
Same options, It would be great to leave interface for both cloud and on-prem users and we can better extend it. I will follow the issue your create and we can discuss more details there
stale commentedon Apr 18, 2020
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
ches commentedon May 7, 2020
@Jeffwan Are you at a point in requirements gathering where we can tighten the focus/scope of this issue, and/or break it out into sub-issues?
I'd like to re-title it for tracking, "AWS support" is quite a broad thing if we're talking about various storage engines, Beam runners, ensuring Helm charts are friendly for any managed Kubernetes service like EKS, etc. It's possible people have done work on some of these areas already, or can do it in parallel to someone else's focus.
Jeffwan commentedon May 12, 2020
@ches
I agree
AWS supportis quite a broad story. This is like a parent issue and try to collect more detail requirements on the AWS support.I think we will have a few phase for this issue.
Phase 1
Add batch store, seems most of the users are looking for Athena. I am check the efforts to implement a new connector for Athena. Even Beam IO doesn't have native integration, I think it's not a problem.
Goal: User will get a workable solution for both online and offline feature.
Phase 2
Both Kafka, Redis have corresponding managed service in AWS, they're Amazon MSK and Amazon ElastiCache. They should work with current feast seamlessly. We need some tutorial for users who like to use managed services and probably provides some cloudformation template to provision AWS resources.
Ingestion currently only supports DirectRunner and DataflowRunner, AWS doesn't have managed Beam services, an alternative for large scale data is needed, either try to use Glue or EMR to run these jobs or use Kubernetes native flink operator solution.
Goal: User can use feast for production grade workloads and can leverage all capability of AWS.
Phase 3
to be determined.
What do you think?
dr3s commentedon May 14, 2020
@Jeffwan I'm working on a batch store and the thing I didn't realize is the client server reliance on GCP storage for data transfer. Unlike the storage API this has not yet been made pluggable. I'm going to open a new issue and start work on this first unless someone else has a branch in progress. I was going to approach it tactically with direct support for s3:// urls. After that I'd like to follow up about a more flexible API.
There is some overlap in needs between Phase 1 and 2 that may be good to draw out. Extracting from Athena PITC features will likely require Beam or some other data processing. It may be worth thinking about how to run beam on flink in Glue/EMR/kubernetes to both process data from athena and kafka.
ches commentedon May 15, 2020
@Jeffwan Sounds good, I understand better where you're coming from now in the overall picture and agree with viewing this issue as an epic that will have child tasks. We could put this issue and those that come "under" it in a project if that's helpful.
I suggest we create a dedicated issue to track Athena implementation as soon as you're concretely moving on that if it's decided, better visibility to Feast users and contributors. Implementing a Beam IO for Athena sounds like a good move to me FWIW.
I think you're right about the biggest challenge being the analog of a Beam Runner you note in your Phase 2. This falls near to #444 (kind of a generalization of #362). I hadn't thought of that issue as posing alternatives to Beam entirely, more about how to cleanly support more choices of Beam Runners, but perhaps you will come to a conclusion that may be the best way you can proceed—if so I think discussion of such a proposal should go to #444.
Thanks for your response and interest in carrying this through.
dr3s commentedon May 30, 2020
@ches @woop We are moving forward with implementation of some of the components necessary to run in AWS. A good first step will be to create an RFC to outline the approach. I started one here: https://docs.google.com/document/d/14ouhFlFiw2OXW5m_esoW0fI9iR7N0TfMd_szi7O2aCk
So far we have noted the following needs:
woop commentedon May 30, 2020
Hi @dr3s,
Excited about this, thanks for kicking off the document!
Let me know if you need help anywhere, especially with RFC reviews or contribution. Otherwise I will see if I can get (3) and (5) ready for you.
[-]Add AWS native support for feast[/-][+]Add AWS native support for Feast[/+]woop commentedon Sep 16, 2020
We have an RFC for Feast on AWS over here: https://docs.google.com/document/d/1snRxVb8ipWZjCiLlfkR4Oc28p7Fkv_UXjvxBFWjRBj4/edit
Feedback welcome.
rceballos98 commentedon Dec 16, 2020
I see AWS support as part of 0.8 release, does this mean it is fully supported now?
From the changelog it seems it is not yet fully there? https://github.com/feast-dev/feast/blob/master/CHANGELOG.md
woop commentedon Dec 21, 2020
AWS support has landed. The only part that is missing is the offline store. We expect the data that you would like to serve to be in S3 in Parquet format, and we will generate training datasets from that data.
woop commentedon Feb 8, 2021
Closing this issue for now. Please see https://docs.feast.dev/getting-started/install-feast/kubernetes-amazon-eks-with-terraform for more details on AWS support. Let's create individual feature requests for specific functionality that we want to add (specific databases like Dynamo for example).
davidshtian commentedon Apr 12, 2021
Not sure the current offline store implementation on AWS is S3 in Parquet format or Hive on EMR? Thanks~