← 输入框|github.com|刷新
Skip to content

How to run / develop for Feast on M1 Macs #2105

Open
Open
How to run / develop for Feast on M1 Macs#2105
Labels
keep-openkind/process

Description

@woop
woop
opened on Dec 3, 2021

Overview

This guide summarizes how both end users and Feast developers can get their M1 Macs set up to use or develop for Feast. pyenv is used in this guide, but isn't absolutely necessary.

Installation Guide

  1. Install pyenv

https://github.com/pyenv/pyenv#homebrew-in-macos

  1. Install dependencies
brew install xz protobuf openssl zlib
  1. Install Python
    (only tested with 3.8.12)
CFLAGS="-I$(brew --prefix xz)/include" LDFLAGS="-L$(brew --prefix xz)/lib" pyenv install 3.8.12
  1. Upgrade Pip and install dependencies
pip install --upgrade pip
pip install cryptography -U

export CFLAGS="-I$(brew --prefix protobuf)/include"
export LDFLAGS="-L$(brew --prefix protobuf)/lib"

# Please set the protobuf version below (we're hardcoding it to 3.17 here)
pip install protobuf==3.17 --force-reinstall --no-deps --install-option="--cpp_implementation"

You can stop here if you are just a Feast user (not a developer). At this point you should be able to pip install feast. Steps (5) and (6) are for developers only

  1. Build and install Feast repository (run from Feast project root)
export LDFLAGS="-L$(brew --prefix freetds)/lib -L$(brew --prefix protobuf)/lib -L$(brew --prefix zlib)/lib -L$(brew --prefix openssl)/lib"
export CPPFLAGS="-I$(brew --prefix protobuf)/include -I$(brew --prefix zlib)/include"
export MYSQLCLIENT_LDFLAGS=$(pkg-config --libs mysqlclient)
export MYSQLCLIENT_CFLAGS=$(pkg-config --cflags mysqlclient)
export GRPC_PYTHON_BUILD_SYSTEM_OPENSSL=1
export GRPC_PYTHON_BUILD_SYSTEM_ZLIB=1

pip install pip-tools
brew install mysql
make install-python-ci-dependencies PYTHON=3.8
  1. Run Feast tests (all tests should pass)
FEAST_USAGE=False IS_TEST=True python -m pytest -n 8 sdk/python/tests

Problems & Solutions

Problem

Lack of _ctypes

Solution

pyenv/pyenv#1768 (comment)

Problem

Could not build wheels for cryptography, which is required to install pyproject.toml-based projects

Solution

pip install --upgrade pip
pip install cryptography -U

Problem

distutils.errors.DistutilsExecError: command 'clang' failed with exit status 1

Solution

export GRPC_PYTHON_BUILD_SYSTEM_OPENSSL=1
export GRPC_PYTHON_BUILD_SYSTEM_ZLIB=1

Problem

ImportError: cannot import name '_message' from 'google.protobuf.pyext' (/opt/homebrew/Caskroom/miniforge/base/envs/intel_py39/lib/python3.9/site-packages/google/protobuf/pyext/__init__.py)

Solution

brew install protobuf
export CFLAGS="-I$(brew --prefix protobuf)/include"
export LDFLAGS="-L$(brew --prefix protobuf)/lib"

# You need
pip install protobuf==3.17 --force-reinstall --no-deps --install-option="--cpp_implementation"

Problem

ModuleNotFoundError: No module named '_lzma'

Solution

brew install xz

CFLAGS="-I$(brew --prefix xz)/include" LDFLAGS="-L$(brew --prefix xz)/lib" pyenv install 3.9.4

Activity

  1. woop
    changed the title [-]How to run/develop Feast on an M1 Mac[/-] [+]How to run or develop for Feast on an M1 Mac[/+] on Dec 3, 2021
  2. github-actions
    assigned
    tsotnet
    on Dec 3, 2021
  3. woop
    unassigned
    tsotnet
    on Dec 3, 2021
  4. woop
    added
    kind/process
    on Dec 3, 2021

  5. MattDelac commented on Dec 13, 2021

    @MattDelac
    MattDelac
    on Dec 13, 2021
    Collaborator

    Thanks @woop, it worked like expected.

    Just needed to reinstall protobuf as you explained

    export CFLAGS="-I$(brew --prefix protobuf)/include"
export LDFLAGS="-L$(brew --prefix protobuf)/lib"

# Please set the protobuf version below (we're hardcoding it to 3.17 here)
pip install protobuf==3.17 --force-reinstall --no-deps --install-option="--cpp_implementation"
  6. judahrand
    pinned this issue on Dec 16, 2021

  7. startakovsky commented on Jan 14, 2022

    @startakovsky
    startakovsky
    on Jan 14, 2022

    Hey @woop using my brand new macbook pro here, M1 chip running Monterey Mac OS. Thank you.

    I can confirm this works also with python 3.10.1. Because my pip freeze was empty at the onset of this, I did simplify my protobuf installation to: pip install protobuf --install-option="--cpp_implementation".

  8. kevjumba commented on Feb 4, 2022

    @kevjumba
    kevjumba
    on Feb 4, 2022
    Collaborator

    Hi,
    Also, people should use the Rosetta M1 Mac plugin.
    I just finished the installation process for the M1 14" Macbook Pro on Mac OSX Monterey 12.1.

    Here are the extra steps I had to perform.

    Fixing Cryptography Installation

    1. Add new dependency brew install rust
    2. Add these exports before hand:
    • export CRYPTOGRAPHY_SUPPRESS_LINK_FLAGS="1"
    • export LDFLAGS="-L/usr/local/opt/openssl/lib"
    • export CPPFLAGS="-I/usr/local/opt/openssl/include"
    1. I changed the installation instructions for cryptography to pip install cryptography==3.4.8 -U (This is the autogenerated version in the requirements from pip-compile.

    Fixing grpc-io installation

    1. Add these exports before running make.
    • CFLAGS="-I/opt/homebrew/opt/openssl/include"
    • LDFLAGS="-L/opt/homebrew/opt/openssl/lib"

    Fixing makefile error below

    Error: Invalid value for '[SRC_FILES]...': Path 'requirements/py-ci-requirements.txt' does not exist. make: *** [install-python-ci-dependencies] Error 2

    • Run export PYTHON=3.8

  9. kevjumba commented on Feb 18, 2022

    @kevjumba
    kevjumba
    on Feb 18, 2022
    Collaborator

    Another issue that I found with setup with java. The protobuf and protc-gen-grpc-java artifacts are not yet available for m1 arm mac osx. In order to fix the artifact not found error when you build make build-java, make test-java and make test-java-integration, change the two artifacts in java/datatypes/pom.xml to

    <protocArtifact>
    com.google.protobuf:protoc:${protoc.version}:exe:osx-x86_64
</protocArtifact>
<pluginId>grpc-java</pluginId>
<pluginArtifact>
    io.grpc:protoc-gen-grpc-java:${grpc.version}:exe:osx-x86_64
</pluginArtifact>
  10. adchia
    changed the title [-]How to run or develop for Feast on an M1 Mac[/-] [+]How to run / develop for Feast on M1 Macs[/+] on Apr 21, 2022
  11. kevjumba
    added
    keep-open
    on Aug 3, 2022

  12. expediamatt commented on Jun 29, 2023

    @expediamatt
    expediamatt
    on Jun 29, 2023

    You might have to change backports-zoneinfo==0.2.1 to backports-zoneinfo==0.2.1;python_version<"3.9" in sdk/python/requirements/py3.8-ci-requirements.txt to make sure it doesn't grab an unworkable copy of backports-zoneinfo.

  13. jeremyary
    unpinned this issue on Feb 7, 2024
  14. jeremyary
    pinned this issue on Feb 7, 2024

  15. dheerajoruganty commented on Feb 15, 2024

    @dheerajoruganty
    dheerajoruganty
    on Feb 15, 2024

    In the most recent pip 24.0 release, the --install-option has been deprecated. As an alternative, you can use
    --config-settings in place of --install-option.

    Here's the modified the command accordingly:
    pip install protobuf --force-reinstall --no-deps --config-settings="--build-option=--cpp_implementation"

  16. tokoko
    unpinned this issue on Jul 11, 2024
  17. tokoko
    pinned this issue on Jul 11, 2024
  18. shuchu
    unpinned this issue on Jul 13, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    keep-openkind/process

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions