Maaz Zindani

Loading the index…

↑↓ move↵ openesc close4072 pages
Servers, nginx and Docker

A Kafka KRaft node cannot register with the controller quorum

Errorthe broker keeps retrying and never finishes startup

Short answer

A KRaft node can only join a quorum it can find and whose cluster ID it shares. Check three things: the controller list in controller.quorum.voters or controller.quorum.bootstrap.servers points at real controller addresses, every node was formatted with the same cluster ID, and a new controller in a dynamic quorum was added with add-controller.

  • Applies to Apache Kafka 4.2, Docker Engine 27
  • Updated
  • By Maaz Zindani, Senior Software Engineer

Fix it fast

  1. 1Find out which cause you have

    Ask the quorum for its state from a node that is running. It shows the cluster ID, the leader, the voters and the observers.

    bin/kafka-metadata-quorum.sh --bootstrap-controller controller1:9093 describe --status
  2. 2Match what you see, then jump to its fix

    1. 1Hostnames in the config do not resolve from inside the container, or ports differ from the controller listenerThe quorum list does not match the real controller addressesFix
    2. 2Each node was formatted separately and the cluster ID shown differs between nodesNodes were formatted with different cluster IDsFix
    3. 3The new controller runs but is an observer, not a voterA new controller was started but never added to a dynamic quorumFix

What does the error look like?

  • the broker keeps retrying and never finishes startup
  • a new controller starts but does not appear among the voters
  • worked on one node, fails when a second or third node is added

Cause 1 of 3

The quorum list does not match the real controller addresses

1Check

In a static quorum every broker and controller lists the id, host and port of all controllers in controller.quorum.voters. In a dynamic quorum that setting must be absent and controller.quorum.bootstrap.servers lists as many controllers as possible. Check which mode you formatted with.

bin/kafka-features.sh --bootstrap-controller controller1:9093 describe
grep -n "controller.quorum" config/server.properties

2Fix

Use one mode on every node. For a static quorum list every controller as id@host:port with the controller listener port; for a dynamic one set controller.quorum.bootstrap.servers instead and drop controller.quorum.voters. In containers or Nomad tasks, use names or addresses the other nodes can reach.

# static quorum
controller.quorum.voters=1@controller1:9093,2@controller2:9093,3@controller3:9093
# dynamic quorum (do not also set voters)
controller.quorum.bootstrap.servers=controller1:9093,controller2:9093,controller3:9093

3Verify

Restart the node and read the quorum state again.

bin/kafka-metadata-quorum.sh --bootstrap-controller controller1:9093 describe --status

Confirmed by kafka.apache.org

Did this fix it?

Cause 2 of 3

Nodes were formatted with different cluster IDs

1Check

Generate the cluster ID once and use the same one when formatting every server. Compare the ID in meta.properties on each node with the one in describe --status.

cat <metadata.log.dir>/meta.properties
bin/kafka-metadata-quorum.sh --bootstrap-controller controller1:9093 describe --status

2Fix

Reformat the node that has a different ID using the cluster ID of the quorum. This erases that node's Kafka metadata, so do it only on a node with no data you need.

KAFKA_CLUSTER_ID="$(bin/kafka-storage.sh random-uuid)"   # once, for a new cluster
bin/kafka-storage.sh format -t $KAFKA_CLUSTER_ID -c config/server.properties

3Verify

The node's ID now matches the quorum and it starts.

bin/kafka-server-start.sh config/server.properties

Confirmed by kafka.apache.org

Did this fix it?

Cause 3 of 3

A new controller was started but never added to a dynamic quorum

1Check

In a dynamic quorum a new controller is provisioned and started, lets replication catch up, and only then is added with add-controller. Until then it is not a voter.

bin/kafka-metadata-quorum.sh --bootstrap-controller controller1:9093 describe --replication

2Fix

Format it with --no-initial-controllers, start it, wait until it has caught up, then run add-controller. Admin client settings such as authentication go in controller.properties.

bin/kafka-storage.sh format -t $KAFKA_CLUSTER_ID -c config/controller.properties --no-initial-controllers
bin/kafka-metadata-quorum.sh --bootstrap-controller controller1:9093 add-controller

3Verify

The new id is listed among the voters.

bin/kafka-metadata-quorum.sh --bootstrap-controller controller1:9093 describe --status

Confirmed by kafka.apache.org

Did this fix it?

How do I stop it coming back?

  • Generate the cluster ID once and keep it in your deployment config, not in each node's start script.
  • Pick static or dynamic quorum at format time and use the same mode on every node.
  • Remove a controller with remove-controller before you shut it down.
  • Kafka does not auto-format blank directories; format each new node on purpose.

Sources

How was this page made?

I drafted this page with AI assistance from the official documents listed under Sources. Every cause above names the document that confirms it. I have not reproduced this fix on a machine of my own yet, so read it as documented guidance, not a test report. Found a mistake? Tell me.

Which stack is behind this page?

Projects of mine built on the same technologies:

Related fixes

Next step

Want this handled for good?

I look after servers, apps and releases end to end. Tell me what keeps breaking.