pip install confluent-kafka fails because librdkafka/rdkafka.h is missing
fatal error: librdkafka/rdkafka.h: No such file or directoryShort answer
The error means pip is compiling confluent-kafka from source, and the compiler cannot find the librdkafka C headers. First check why no prebuilt wheel was used. Then install the librdkafka development package for your OS (librdkafka-dev on Debian or Ubuntu, librdkafka-devel on RHEL-like systems, brew on macOS) and install again.
Fix it fast
1Find out which cause you have
Ask pip to use only prebuilt wheels. It will refuse with a clear message if no wheel fits your Python and platform, which confirms that the failed run was a source build.
python3 -m pip install --only-binary confluent-kafka confluent-kafka2Match what you see, then jump to its fix
- 1
The build ran on Debian, Ubuntu or a Dockerfile based on them, and the compiler stops at rdkafka.hDebian or Ubuntu (including a Docker image) has no librdkafka headersFix - 2
The same missing header on a yum or dnf based systemRHEL, CentOS or Fedora is missing librdkafka-develFix - 3
The error appears on a Mac when pip builds from sourcemacOS has no librdkafka from HomebrewFix
- 1
What does the error look like?
fatal error: librdkafka/rdkafka.h: No such file or directorypip stops while building confluent-kafka from source
Cause 1 of 3
Debian or Ubuntu (including a Docker image) has no librdkafka headers
1Check
Confluent's INSTALL.md lists building from source for cases where no wheel is used, and starts with installing librdkafka. See whether the dev package is present.
dpkg -s librdkafka-dev 2>&1 | head -n 3
ls /usr/include/librdkafka/rdkafka.h2Fix
Install the development package, then force a source build against it.
apt install -y librdkafka-dev
python3 -m pip install --no-binary confluent-kafka confluent-kafka3Verify
The module imports without an error.
python3 -c "import confluent_kafka"Confirmed by github.com
Cause 2 of 3
RHEL, CentOS or Fedora is missing librdkafka-devel
1Check
On these systems the header comes from the devel package. Query it with rpm.
rpm -q librdkafka-devel2Fix
Install the package named in the install guide, then build confluent-kafka from source.
yum install -y librdkafka-devel
python3 -m pip install --no-binary confluent-kafka confluent-kafka3Verify
The import works.
python3 -c "import confluent_kafka"Confirmed by github.com
Cause 3 of 3
macOS has no librdkafka from Homebrew
1Check
Homebrew is the install route the guide gives for macOS. Check whether the formula is installed.
brew list librdkafka2Fix
Install the formula and build again.
brew install librdkafka
python3 -m pip install --no-binary confluent-kafka confluent-kafka3Verify
The import works.
python3 -c "import confluent_kafka"Confirmed by github.com
How do I stop it coming back?
- In a Dockerfile, install librdkafka-dev in an earlier RUN than the pip install.
- Pin the confluent-kafka version in requirements.txt so a new release does not change what gets built.
- Use the same Python version in development and in the image.
Sources
How was this page made?
I drafted this page with AI assistance from the official documents listed under Sources. Every cause above names the document that confirms it. I have not reproduced this fix on a machine of my own yet, so read it as documented guidance, not a test report. Found a mistake? Tell me.
Which stack is behind this page?
Projects of mine built on the same technologies:
- TaxiDexMy own multi-tenant taxi dispatch SaaS: API, dispatcher portal, driver and passenger apps, and the servers underneath.
- Dispatch ProA real-time logistics platform: Flutter apps, Spring Boot services and a Kafka event pipeline on AWS.
- stegnor.comMy company site: 804 pages from a Python and Jinja generator, with 15 free browser tools.
Related fixes
Servers
Docker build fails with apk: not found in a Python or Ubuntu image
/bin/sh: 1: apk: not found
apk is the package manager of Alpine Linux only.
2 causes, each with a fixServers
You cannot pip install packages inside an Ubuntu Docker container
error: externally-managed-environment
If pip is missing, install the python3-pip package with apt, in the same RUN as apt-get update.
3 causes, each with a fixServers
pip install psycopg2 fails while building the Docker image
libpq-fe.h: No such file or directory
The plain psycopg2 package compiles a C extension, so the image needs a compiler, the Python headers, the libpq headers and pg_config.
3 causes, each with a fixServers
Apache logs SIGWINCH and exits on its own
the httpd container stops although nothing crashed
SIGWINCH is Apache's graceful-stop signal.
2 causes, each with a fixServers
sam local fails: Running AWS SAM projects locally requires Docker
Cannot connect to the Docker daemon. Is the docker daemon running on this host?
sam local start-api and sam local invoke run your function inside a Docker container, so SAM stops if it cannot talk to Docker.
3 causes, each with a fixServers
dnf fails to download repo metadata in a CentOS 8 Docker build
The same Dockerfile built fine before 2022
CentOS Linux 8 reached end of life on 31 December 2021 and its content left the mirrors.
2 causes, each with a fix