pip tries to compile tokenizers and fails with a Rust or cargo error
cargo rustc failed with code 101Short answer
tokenizers is written in Rust. pip only compiles it when no prebuilt wheel fits your Python version and platform, and then it needs a Rust toolchain. First check you run Python 3.10 or newer on a platform with published wheels. If you must compile, install Rust with rustup and retry.
Fix it fast
1Find out which cause you have
Ask pip to use wheels only. If a wheel exists for your Python and platform it installs at once; if not, pip says why, and that tells you which cause applies.
python --version python -m pip install --only-binary=:all: tokenizers2Match what you see, then jump to its fix
- 1
pip --only-binary reports no matching distribution, and python --version prints 3.9 or lowerYour Python is older than the version the wheels targetFix - 2
--only-binary fails on Python 3.10 or newer tooNo wheel is published for your platformFix - 3
cargo or rustc: command not found, or a compiler error from rustc itselfBuilding from source needs Rust and it is missing or oldFix
- 1
What does the error look like?
cargo rustc failed with code 101Could not build wheels for tokenizers, which is required to install pyproject.toml-based projects
Cause 1 of 3
Your Python is older than the version the wheels target
1Check
PyPI lists tokenizers 0.23 as requiring Python 3.10 or newer, and the Hugging Face docs say it is tested on Python 3.10+. Older interpreters either get an old release or fall back to the source package.
python --version2Fix
Use a Python 3.10 or newer interpreter, for example in a fresh virtual environment, and install again.
python3.12 -m venv .venv
. .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install tokenizers3Verify
The install finishes in seconds without compiling.
python -c "import tokenizers; print(tokenizers.__version__)"Confirmed by pypi.org
Cause 2 of 3
No wheel is published for your platform
1Check
PyPI lists wheels for Windows, macOS, manylinux glibc 2.17+ and musllinux 1.2+ on common CPUs. A platform outside that list (or an operating system whose C library is older than the tag) gets the source archive instead.
uname -m
ldd --version | head -n 12Fix
Move to a base image or system that a wheel exists for (a current Debian or Ubuntu, not an old one), or install Rust and build from source as in the next cause.
# Dockerfile
FROM python:3.12-slim3Verify
Install again; pip downloads a .whl file instead of a .tar.gz.
python -m pip install -v tokenizers 2>&1 | grep -iE "\.whl|\.tar\.gz"Confirmed by pypi.org
Cause 3 of 3
Building from source needs Rust and it is missing or old
1Check
Compiling tokenizers needs the Rust language installed. Check the tools exist and are current.
cargo --version
rustc --version2Fix
Install Rust with rustup. If it is already installed, update it. Open a new shell afterwards so PATH picks up cargo.
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
rustup update3Verify
cargo answers, then the pip install compiles and finishes.
cargo --version
python -m pip install tokenizersConfirmed by huggingface.co
How do I stop it coming back?
- Pin the Python minor version in your Dockerfile and CI so the wheel that worked keeps working.
- Use pip install --only-binary=:all: for heavy packages in images, so a missing wheel fails loudly instead of starting a long compile.
- Keep pip itself current in the build step.
Sources
How was this page made?
I drafted this page with AI assistance from the official documents listed under Sources. Every cause above names the document that confirms it. I have not reproduced this fix on a machine of my own yet, so read it as documented guidance, not a test report. Found a mistake? Tell me.
Which stack is behind this page?
Projects of mine built on the same technologies:
- stegnor.comMy company site: 804 pages from a Python and Jinja generator, with 15 free browser tools.
- MenuloA restaurant discovery platform in Germany, kept current by a Python ingestion pipeline.
- ShopGoAn e-commerce aggregator with a Python scraping engine behind its catalog, past 10K Google Play downloads.
Related fixes
Servers
pip install confluent-kafka fails because librdkafka/rdkafka.h is missing
pip stops while building confluent-kafka from source
The error means pip is compiling confluent-kafka from source, and the compiler cannot find the librdkafka C headers.
3 causes, each with a fixServers
cv2.imshow raises 'not implemented' because the wrong OpenCV package is installed
cv2.imshow() works in one environment and fails in another
The headless OpenCV packages are built without any GUI code, so cv2.imshow cannot open a window.
2 causes, each with a fixServers
pip install Cython succeeded, but the shell cannot find cythonize
a pip warning that a script was installed to a directory outside PATH
pip put the cythonize script in a folder your shell does not search.
3 causes, each with a fixServers
Django returns a 500 behind Gunicorn and the traceback is nowhere
The traceback shows locally with runserver but not under Gunicorn
With DEBUG set to False, Django's default logging does not print errors to the console.
2 causes, each with a fixServers
Docker build fails with apk: not found in a Python or Ubuntu image
/bin/sh: 1: apk: not found
apk is the package manager of Alpine Linux only.
2 causes, each with a fixServers
You cannot pip install packages inside an Ubuntu Docker container
error: externally-managed-environment
If pip is missing, install the python3-pip package with apt, in the same RUN as apt-get update.
3 causes, each with a fix