A Galera node will not start: gcomm backend connection failed with error 110
gcomm backend connection failed: 110Short answer
Error 110 is a timeout: the node tried to join the addresses in wsrep_cluster_address and reached no live cluster. Either nothing is running to join, the cluster ports are blocked between nodes, the address uses the wrong port, or the whole cluster went down and the right node was not bootstrapped first.
Fix it fast
1Find out which cause you have
Read what the node logged just before it gave up, and confirm what it was told to join.
sudo journalctl -u mariadb.service -n 50 --no-pager mariadbd --help --verbose | grep wsrep-cluster-address2Match what you see, then jump to its fix
- 1
Every node lists the others in gcomm:// and every node times outNo node is running to join, so nobody bootstrapped the clusterFix - 2
Timeouts on one node while the others look healthy, often after adding a host or a new firewallThe Galera ports are blocked between the nodesFix - 3
The first node starts, but this node never connects to itwsrep_cluster_address has the wrong port or an unreachable hostFix - 4
Nothing runs after a full outage, and the node you started first times outThe whole cluster was down and the wrong node went firstFix
- 1
What does the error look like?
gcomm backend connection failed: 110Unit mariadb.service entered failed state.
Cause 1 of 4
No node is running to join, so nobody bootstrapped the cluster
1Check
A node started normally only joins an existing cluster. If all nodes are down, or this is the first start ever, there is nothing to join.
sudo systemctl status mariadb.service2Fix
Start exactly one node as the first node of a new cluster, then start the others normally. Do not keep the bootstrap option on afterwards: it creates a new cluster UUID.
# on the first node only
sudo galera_new_cluster
# on each other node
sudo systemctl start mariadb3Verify
Ask the first node how many members it sees.
mariadb -e "SHOW GLOBAL STATUS LIKE 'wsrep_cluster_size'"Confirmed by mariadb.com
Cause 2 of 4
The Galera ports are blocked between the nodes
1Check
Galera needs port 4567 for replication traffic, 4568 for incremental state transfer and 4444 for the other state transfer methods, as well as 3306 for clients. Test the replication port from the failing node to another node.
nc -zv 192.168.0.1 45672Fix
Allow TCP on 4567, 4568 and 4444 (and 3306 for clients) in both directions between all cluster nodes, in whatever firewall or cloud security group the hosts use.
# firewall rule set, for every node pair:
# tcp 3306, 4567, 4568, 44443Verify
The port test succeeds, then restart the node.
nc -zv 192.168.0.1 4567
sudo systemctl restart mariadbConfirmed by mariadb.com
Cause 3 of 4
wsrep_cluster_address has the wrong port or an unreachable host
1Check
A port written in gcomm:// is the Galera port, not the MariaDB port, so :3306 there points at the wrong service. The docs advise listing all possible members.
grep -R "wsrep_cluster_address" /etc/mysql/ /etc/my.cnf.d/ 2>/dev/null2Fix
List every node by address, with no port unless you changed the Galera port.
[mariadb]
wsrep_cluster_address=gcomm://192.168.0.1,192.168.0.2,192.168.0.33Verify
Restart the node and check it joined.
sudo systemctl restart mariadb
mariadb -e "SHOW GLOBAL STATUS LIKE 'wsrep_cluster_size'"Confirmed by mariadb.com
Cause 4 of 4
The whole cluster was down and the wrong node went first
1Check
After all nodes stop, the node to start from is the one with the highest seqno in its grastate.dat, found in the data directory.
sudo cat /var/lib/mysql/grastate.dat2Fix
Bootstrap the node with the highest seqno, then start the other nodes normally so they sync from it.
# on the node with the highest seqno
sudo galera_new_cluster
# then on each other node
sudo systemctl start mariadb3Verify
The cluster size matches the number of nodes.
mariadb -e "SHOW GLOBAL STATUS LIKE 'wsrep_cluster_size'"Confirmed by mariadb.com
How do I stop it coming back?
- Write down which node was last to stop before a planned full shutdown, and start that node first.
- Never hard-code an empty gcomm:// address in a config file.
- Keep one firewall rule set for all cluster nodes so a new host cannot be missed.
- Monitor wsrep_cluster_size and alert when it drops below the node count.
Sources
How was this page made?
I drafted this page with AI assistance from the official documents listed under Sources. Every cause above names the document that confirms it. I have not reproduced this fix on a machine of my own yet, so read it as documented guidance, not a test report. Found a mistake? Tell me.
Which stack is behind this page?
Projects of mine built on the same technologies:
- TaxiDexMy own multi-tenant taxi dispatch SaaS: API, dispatcher portal, driver and passenger apps, and the servers underneath.
- QistLockA NestJS and React SaaS driving an Android Device Owner agent that locks financed phones when payments lapse.
- ShopGoAn e-commerce aggregator with a Python scraping engine behind its catalog, past 10K Google Play downloads.
Related fixes
Databases
Installing the Airflow MySQL provider fails on pkg-config or mysqlclient
Building the mysqlclient wheel fails during the install
The Airflow MySQL provider depends on mysqlclient, which pip compiles against the MySQL C client library.
2 causes, each with a fixDatabases
Compiling a C or C++ program that includes mysql.h fails
undefined reference to `mysql_init'
The compiler only finds mysql.h when the MySQL client development files are installed and their folder is on the include path.
3 causes, each with a fixDatabases
An EC2 instance cannot reach its RDS database
Could not open connection to the host, on port <port>: Connect failed
RDS denies all access until a security group allows it.
3 causes, each with a fixDatabases
Locked out of a fresh MariaDB install as root on Mac or Linux
ERROR 1045 (28000): Access denied for user 'root'@'localhost' (using password: NO)
A fresh MariaDB root account is not a normal password account.
3 causes, each with a fixDatabases
An Aria table fails with 'Read page with wrong checksum' (error 1030, code 176)
ERROR 1030 (HY000): Got error 176 "Read page with wrong checksum" from storage engine Aria
Code 176 means MariaDB's Aria engine read a data or index page whose checksum does not match, so that table file is damaged.
3 causes, each with a fixDatabases
The client cannot open a TCP connection to MariaDB or MySQL (error 2003)
ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1:3306' (111)
Error 2003 means the client tried a TCP connection to the host and port and nobody answered.
4 causes, each with a fix