Maaz Zindani

Loading the index…

↑↓ move↵ openesc close4072 pages
Servers, nginx and Docker

pip tries to compile tokenizers and fails with a Rust or cargo error

Errorcargo rustc failed with code 101

Short answer

tokenizers is written in Rust. pip only compiles it when no prebuilt wheel fits your Python version and platform, and then it needs a Rust toolchain. First check you run Python 3.10 or newer on a platform with published wheels. If you must compile, install Rust with rustup and retry.

  • Applies to tokenizers 0.23, pip 24 and 25, Python 3.10 to 3.13, Linux containers
  • Updated
  • By Maaz Zindani, Senior Software Engineer

Fix it fast

  1. 1Find out which cause you have

    Ask pip to use wheels only. If a wheel exists for your Python and platform it installs at once; if not, pip says why, and that tells you which cause applies.

    python --version
    python -m pip install --only-binary=:all: tokenizers
  2. 2Match what you see, then jump to its fix

    1. 1pip --only-binary reports no matching distribution, and python --version prints 3.9 or lowerYour Python is older than the version the wheels targetFix
    2. 2--only-binary fails on Python 3.10 or newer tooNo wheel is published for your platformFix
    3. 3cargo or rustc: command not found, or a compiler error from rustc itselfBuilding from source needs Rust and it is missing or oldFix

What does the error look like?

  • cargo rustc failed with code 101
  • Could not build wheels for tokenizers, which is required to install pyproject.toml-based projects

Cause 1 of 3

Your Python is older than the version the wheels target

1Check

PyPI lists tokenizers 0.23 as requiring Python 3.10 or newer, and the Hugging Face docs say it is tested on Python 3.10+. Older interpreters either get an old release or fall back to the source package.

python --version

2Fix

Use a Python 3.10 or newer interpreter, for example in a fresh virtual environment, and install again.

python3.12 -m venv .venv
. .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install tokenizers

3Verify

The install finishes in seconds without compiling.

python -c "import tokenizers; print(tokenizers.__version__)"

Confirmed by pypi.org

Did this fix it?

Cause 2 of 3

No wheel is published for your platform

1Check

PyPI lists wheels for Windows, macOS, manylinux glibc 2.17+ and musllinux 1.2+ on common CPUs. A platform outside that list (or an operating system whose C library is older than the tag) gets the source archive instead.

uname -m
ldd --version | head -n 1

2Fix

Move to a base image or system that a wheel exists for (a current Debian or Ubuntu, not an old one), or install Rust and build from source as in the next cause.

# Dockerfile
FROM python:3.12-slim

3Verify

Install again; pip downloads a .whl file instead of a .tar.gz.

python -m pip install -v tokenizers 2>&1 | grep -iE "\.whl|\.tar\.gz"

Confirmed by pypi.org

Did this fix it?

Cause 3 of 3

Building from source needs Rust and it is missing or old

1Check

Compiling tokenizers needs the Rust language installed. Check the tools exist and are current.

cargo --version
rustc --version

2Fix

Install Rust with rustup. If it is already installed, update it. Open a new shell afterwards so PATH picks up cargo.

curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
rustup update

3Verify

cargo answers, then the pip install compiles and finishes.

cargo --version
python -m pip install tokenizers

Confirmed by huggingface.co

Did this fix it?

How do I stop it coming back?

  • Pin the Python minor version in your Dockerfile and CI so the wheel that worked keeps working.
  • Use pip install --only-binary=:all: for heavy packages in images, so a missing wheel fails loudly instead of starting a long compile.
  • Keep pip itself current in the build step.

Sources

How was this page made?

I drafted this page with AI assistance from the official documents listed under Sources. Every cause above names the document that confirms it. I have not reproduced this fix on a machine of my own yet, so read it as documented guidance, not a test report. Found a mistake? Tell me.

Which stack is behind this page?

Projects of mine built on the same technologies:

Related fixes

Next step

Want this handled for good?

I look after servers, apps and releases end to end. Tell me what keeps breaking.