A Kafka KRaft node cannot register with the controller quorum
the broker keeps retrying and never finishes startupShort answer
A KRaft node can only join a quorum it can find and whose cluster ID it shares. Check three things: the controller list in controller.quorum.voters or controller.quorum.bootstrap.servers points at real controller addresses, every node was formatted with the same cluster ID, and a new controller in a dynamic quorum was added with add-controller.
Fix it fast
1Find out which cause you have
Ask the quorum for its state from a node that is running. It shows the cluster ID, the leader, the voters and the observers.
bin/kafka-metadata-quorum.sh --bootstrap-controller controller1:9093 describe --status2Match what you see, then jump to its fix
- 1
Hostnames in the config do not resolve from inside the container, or ports differ from the controller listenerThe quorum list does not match the real controller addressesFix - 2
Each node was formatted separately and the cluster ID shown differs between nodesNodes were formatted with different cluster IDsFix - 3
The new controller runs but is an observer, not a voterA new controller was started but never added to a dynamic quorumFix
- 1
What does the error look like?
the broker keeps retrying and never finishes startupa new controller starts but does not appear among the votersworked on one node, fails when a second or third node is added
Cause 1 of 3
The quorum list does not match the real controller addresses
1Check
In a static quorum every broker and controller lists the id, host and port of all controllers in controller.quorum.voters. In a dynamic quorum that setting must be absent and controller.quorum.bootstrap.servers lists as many controllers as possible. Check which mode you formatted with.
bin/kafka-features.sh --bootstrap-controller controller1:9093 describe
grep -n "controller.quorum" config/server.properties2Fix
Use one mode on every node. For a static quorum list every controller as id@host:port with the controller listener port; for a dynamic one set controller.quorum.bootstrap.servers instead and drop controller.quorum.voters. In containers or Nomad tasks, use names or addresses the other nodes can reach.
# static quorum
controller.quorum.voters=1@controller1:9093,2@controller2:9093,3@controller3:9093
# dynamic quorum (do not also set voters)
controller.quorum.bootstrap.servers=controller1:9093,controller2:9093,controller3:90933Verify
Restart the node and read the quorum state again.
bin/kafka-metadata-quorum.sh --bootstrap-controller controller1:9093 describe --statusConfirmed by kafka.apache.org
Cause 2 of 3
Nodes were formatted with different cluster IDs
1Check
Generate the cluster ID once and use the same one when formatting every server. Compare the ID in meta.properties on each node with the one in describe --status.
cat <metadata.log.dir>/meta.properties
bin/kafka-metadata-quorum.sh --bootstrap-controller controller1:9093 describe --status2Fix
Reformat the node that has a different ID using the cluster ID of the quorum. This erases that node's Kafka metadata, so do it only on a node with no data you need.
KAFKA_CLUSTER_ID="$(bin/kafka-storage.sh random-uuid)" # once, for a new cluster
bin/kafka-storage.sh format -t $KAFKA_CLUSTER_ID -c config/server.properties3Verify
The node's ID now matches the quorum and it starts.
bin/kafka-server-start.sh config/server.propertiesConfirmed by kafka.apache.org
Cause 3 of 3
A new controller was started but never added to a dynamic quorum
1Check
In a dynamic quorum a new controller is provisioned and started, lets replication catch up, and only then is added with add-controller. Until then it is not a voter.
bin/kafka-metadata-quorum.sh --bootstrap-controller controller1:9093 describe --replication2Fix
Format it with --no-initial-controllers, start it, wait until it has caught up, then run add-controller. Admin client settings such as authentication go in controller.properties.
bin/kafka-storage.sh format -t $KAFKA_CLUSTER_ID -c config/controller.properties --no-initial-controllers
bin/kafka-metadata-quorum.sh --bootstrap-controller controller1:9093 add-controller3Verify
The new id is listed among the voters.
bin/kafka-metadata-quorum.sh --bootstrap-controller controller1:9093 describe --statusConfirmed by kafka.apache.org
How do I stop it coming back?
- Generate the cluster ID once and keep it in your deployment config, not in each node's start script.
- Pick static or dynamic quorum at format time and use the same mode on every node.
- Remove a controller with remove-controller before you shut it down.
- Kafka does not auto-format blank directories; format each new node on purpose.
Sources
How was this page made?
I drafted this page with AI assistance from the official documents listed under Sources. Every cause above names the document that confirms it. I have not reproduced this fix on a machine of my own yet, so read it as documented guidance, not a test report. Found a mistake? Tell me.
Which stack is behind this page?
Projects of mine built on the same technologies:
Related fixes
Servers
Creating a Kafka topic against a Docker broker times out waiting for a node assignment
Timed out waiting for a node assignment
The client could not open a working connection to any broker.
2 causes, each with a fixServers
apk cannot find py-pip in an Alpine Docker image
py-pip (no such package):
On current Alpine releases there is no package called py-pip.
2 causes, each with a fixServers
ng serve runs in the container but localhost:4200 does not open
The browser cannot open localhost:4200 although ng serve is running in the container
ng serve listens on localhost by default, which inside a container is the container itself.
3 causes, each with a fixServers
Angular routes return 404 from nginx when you refresh or open a link
The home page loads, but refreshing /some/route shows an nginx 404
Angular routes like /users/42 are not files, so nginx looks for them on disk and returns 404.
3 causes, each with a fixServers
AH00558: Apache in Docker cannot reliably determine its fully qualified domain name
Set the 'ServerName' directive globally to suppress this message
AH00558 is a warning, not a crash.
2 causes, each with a fixServers
Apache in a Docker container logs AH01114 when it proxies to a backend
AH01114: HTTP: failed to make connection to backend: api
Apache's proxy worker could not open a connection to the address in ProxyPass.
5 causes, each with a fix