Maaz Zindani

Loading the index…

↑↓ move↵ openesc close4072 pages
Servers, nginx and Docker

pip install confluent-kafka fails because librdkafka/rdkafka.h is missing

Errorfatal error: librdkafka/rdkafka.h: No such file or directory

Short answer

The error means pip is compiling confluent-kafka from source, and the compiler cannot find the librdkafka C headers. First check why no prebuilt wheel was used. Then install the librdkafka development package for your OS (librdkafka-dev on Debian or Ubuntu, librdkafka-devel on RHEL-like systems, brew on macOS) and install again.

  • Applies to confluent-kafka 2.x, Python 3.12, Debian 12 and Ubuntu 24.04, Docker
  • Updated
  • By Maaz Zindani, Senior Software Engineer

Fix it fast

  1. 1Find out which cause you have

    Ask pip to use only prebuilt wheels. It will refuse with a clear message if no wheel fits your Python and platform, which confirms that the failed run was a source build.

    python3 -m pip install --only-binary confluent-kafka confluent-kafka
  2. 2Match what you see, then jump to its fix

    1. 1The build ran on Debian, Ubuntu or a Dockerfile based on them, and the compiler stops at rdkafka.hDebian or Ubuntu (including a Docker image) has no librdkafka headersFix
    2. 2The same missing header on a yum or dnf based systemRHEL, CentOS or Fedora is missing librdkafka-develFix
    3. 3The error appears on a Mac when pip builds from sourcemacOS has no librdkafka from HomebrewFix

What does the error look like?

  • fatal error: librdkafka/rdkafka.h: No such file or directory
  • pip stops while building confluent-kafka from source

Cause 1 of 3

Debian or Ubuntu (including a Docker image) has no librdkafka headers

1Check

Confluent's INSTALL.md lists building from source for cases where no wheel is used, and starts with installing librdkafka. See whether the dev package is present.

dpkg -s librdkafka-dev 2>&1 | head -n 3
ls /usr/include/librdkafka/rdkafka.h

2Fix

Install the development package, then force a source build against it.

apt install -y librdkafka-dev
python3 -m pip install --no-binary confluent-kafka confluent-kafka

3Verify

The module imports without an error.

python3 -c "import confluent_kafka"

Confirmed by github.com

Did this fix it?

Cause 2 of 3

RHEL, CentOS or Fedora is missing librdkafka-devel

1Check

On these systems the header comes from the devel package. Query it with rpm.

rpm -q librdkafka-devel

2Fix

Install the package named in the install guide, then build confluent-kafka from source.

yum install -y librdkafka-devel
python3 -m pip install --no-binary confluent-kafka confluent-kafka

3Verify

The import works.

python3 -c "import confluent_kafka"

Confirmed by github.com

Did this fix it?

Cause 3 of 3

macOS has no librdkafka from Homebrew

1Check

Homebrew is the install route the guide gives for macOS. Check whether the formula is installed.

brew list librdkafka

2Fix

Install the formula and build again.

brew install librdkafka
python3 -m pip install --no-binary confluent-kafka confluent-kafka

3Verify

The import works.

python3 -c "import confluent_kafka"

Confirmed by github.com

Did this fix it?

How do I stop it coming back?

  • In a Dockerfile, install librdkafka-dev in an earlier RUN than the pip install.
  • Pin the confluent-kafka version in requirements.txt so a new release does not change what gets built.
  • Use the same Python version in development and in the image.

Sources

How was this page made?

I drafted this page with AI assistance from the official documents listed under Sources. Every cause above names the document that confirms it. I have not reproduced this fix on a machine of my own yet, so read it as documented guidance, not a test report. Found a mistake? Tell me.

Which stack is behind this page?

Projects of mine built on the same technologies:

Related fixes

Next step

Want this handled for good?

I look after servers, apps and releases end to end. Tell me what keeps breaking.