Overview

This is a guide on setting up huggingface transformers on a MacBook Air with Apple Silicon (M1).

Install

Python

Tensorflow

NOTE: Tensorflow v2.5 natively supports M1. NOTE:tensorflow-macos, tensorflow-metal are only available on Python 3.8-3.9.

# only 3.8-3.9 supported
python_version=$(python -c "import platform; print(platform.python_version())")
conda create -n [NAME] python=$python_version
conda activate [NAME]
conda install -c apple tensorflow-deps
# conda install -c apple tensorflow-deps==2.6.0

# Pip
python -m pip install tensorflow-macos
python -m pip install tensorflow-metal

# Homebrew deps
brew install libjpeg
conda install -y matplotlib jupyterlab

Test Tensorflow

To test whether tensorflow is working:

(tensorflow) aj@computer ~ % python
Python 3.9.5 | packaged by conda-forge | (default, Oct 19 2021, 17:32:20)
[Clang 11.1.0 ] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import tensorflow as tf
Init Plugin
Init Graph Optimizer
Init Kernel
>>> print(tf.__version__)
2.8.0
>>>

Tokenizers

Installation — tokenizers documentation

# install Rust
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

# activate conda
conda activate [NAME]

# in a new terminal:
git clone https://github.com/huggingface/tokenizers
cd tokenizers/bindings/python
pip install setuptools_rust
python setup.py install

Transformers

conda activate tensorflow
pip install git+https://github.com/huggingface/transformers

Hugging Face Dataset

Installation — datasets 1.18.0 documentation

conda install -c huggingface -c conda-forge datasets

Troubleshooting

MemoryError: Cannot allocate write+execute memory for ffi.callback()

https://github.com/dbt-labs/dbt-core/issues/3162#issuecomment-1011600708

To fix:

brew install libffi

Download wheel

Then install the wheel:

pip install --force-reinstall ~/Downloads/cffi-1.15.0-cp39-cp39-macosx_11_0_arm64.whl

Cannot install tensorflow-macos or tensorflow-metal

If pip cannot find these packages, you may not be on a supported Python version. As of this writing, only Python 3.8-3.9 are supported, so switch to one of those versions.

References

Metadata