Maaz Zindani

Loading the index…

↑↓ move↵ openesc close4072 pages
Databases and Redis

A Galera node will not start: gcomm backend connection failed with error 110

Errorgcomm backend connection failed: 110

Short answer

Error 110 is a timeout: the node tried to join the addresses in wsrep_cluster_address and reached no live cluster. Either nothing is running to join, the cluster ports are blocked between nodes, the address uses the wrong port, or the whole cluster went down and the right node was not bootstrapped first.

  • Applies to MariaDB 10.11, MariaDB 11.4, Galera 4, Ubuntu 24.04
  • Updated
  • By Maaz Zindani, Senior Software Engineer

Fix it fast

  1. 1Find out which cause you have

    Read what the node logged just before it gave up, and confirm what it was told to join.

    sudo journalctl -u mariadb.service -n 50 --no-pager
    mariadbd --help --verbose | grep wsrep-cluster-address
  2. 2Match what you see, then jump to its fix

    1. 1Every node lists the others in gcomm:// and every node times outNo node is running to join, so nobody bootstrapped the clusterFix
    2. 2Timeouts on one node while the others look healthy, often after adding a host or a new firewallThe Galera ports are blocked between the nodesFix
    3. 3The first node starts, but this node never connects to itwsrep_cluster_address has the wrong port or an unreachable hostFix
    4. 4Nothing runs after a full outage, and the node you started first times outThe whole cluster was down and the wrong node went firstFix

What does the error look like?

  • gcomm backend connection failed: 110
  • Unit mariadb.service entered failed state.

Cause 1 of 4

No node is running to join, so nobody bootstrapped the cluster

1Check

A node started normally only joins an existing cluster. If all nodes are down, or this is the first start ever, there is nothing to join.

sudo systemctl status mariadb.service

2Fix

Start exactly one node as the first node of a new cluster, then start the others normally. Do not keep the bootstrap option on afterwards: it creates a new cluster UUID.

# on the first node only
sudo galera_new_cluster
# on each other node
sudo systemctl start mariadb

3Verify

Ask the first node how many members it sees.

mariadb -e "SHOW GLOBAL STATUS LIKE 'wsrep_cluster_size'"

Confirmed by mariadb.com

Did this fix it?

Cause 2 of 4

The Galera ports are blocked between the nodes

1Check

Galera needs port 4567 for replication traffic, 4568 for incremental state transfer and 4444 for the other state transfer methods, as well as 3306 for clients. Test the replication port from the failing node to another node.

nc -zv 192.168.0.1 4567

2Fix

Allow TCP on 4567, 4568 and 4444 (and 3306 for clients) in both directions between all cluster nodes, in whatever firewall or cloud security group the hosts use.

# firewall rule set, for every node pair:
#   tcp 3306, 4567, 4568, 4444

3Verify

The port test succeeds, then restart the node.

nc -zv 192.168.0.1 4567
sudo systemctl restart mariadb

Confirmed by mariadb.com

Did this fix it?

Cause 3 of 4

wsrep_cluster_address has the wrong port or an unreachable host

1Check

A port written in gcomm:// is the Galera port, not the MariaDB port, so :3306 there points at the wrong service. The docs advise listing all possible members.

grep -R "wsrep_cluster_address" /etc/mysql/ /etc/my.cnf.d/ 2>/dev/null

2Fix

List every node by address, with no port unless you changed the Galera port.

[mariadb]
wsrep_cluster_address=gcomm://192.168.0.1,192.168.0.2,192.168.0.3

3Verify

Restart the node and check it joined.

sudo systemctl restart mariadb
mariadb -e "SHOW GLOBAL STATUS LIKE 'wsrep_cluster_size'"

Confirmed by mariadb.com

Did this fix it?

Cause 4 of 4

The whole cluster was down and the wrong node went first

1Check

After all nodes stop, the node to start from is the one with the highest seqno in its grastate.dat, found in the data directory.

sudo cat /var/lib/mysql/grastate.dat

2Fix

Bootstrap the node with the highest seqno, then start the other nodes normally so they sync from it.

# on the node with the highest seqno
sudo galera_new_cluster
# then on each other node
sudo systemctl start mariadb

3Verify

The cluster size matches the number of nodes.

mariadb -e "SHOW GLOBAL STATUS LIKE 'wsrep_cluster_size'"

Confirmed by mariadb.com

Did this fix it?

How do I stop it coming back?

  • Write down which node was last to stop before a planned full shutdown, and start that node first.
  • Never hard-code an empty gcomm:// address in a config file.
  • Keep one firewall rule set for all cluster nodes so a new host cannot be missed.
  • Monitor wsrep_cluster_size and alert when it drops below the node count.

Sources

How was this page made?

I drafted this page with AI assistance from the official documents listed under Sources. Every cause above names the document that confirms it. I have not reproduced this fix on a machine of my own yet, so read it as documented guidance, not a test report. Found a mistake? Tell me.

Which stack is behind this page?

Projects of mine built on the same technologies:

Related fixes

Next step

Want this handled for good?

I look after servers, apps and releases end to end. Tell me what keeps breaking.