Diagnosing and Recovering from Split‑Brain in MariaDB Galera Cluster

Recognizable Condition

Applications report stale reads, write timeouts, or errors such as "Lost connection" while the cluster status shows a value other than Primary for wsrep_cluster_status (e.g., Non-Primary, Disconnected, or Lost) and wsrep_ready is OFF.

Cause and Symptom Reference

Ordered Diagnostic Checks

1. Verify network connectivity: ping node‑address and telnet node‑address 4567 from each host.
2. Check cluster status: SHOW STATUS LIKE 'wsrep_cluster_status'; SHOW STATUS LIKE 'wsrep_ready';
3. Inspect the MariaDB error log for messages containing split brain, gcache, or state transfer.
4. Confirm gcache size adequacy: SHOW GLOBAL STATUS LIKE 'wsrep_gcache_size'; compare with recent write volume.
5. If state snapshot transfer (SST) is needed, verify donor availability: SHOW STATUS LIKE 'wsrep_local_state_comment';

Fixes Tied to Findings

• Network partition restored: after fixing routing/firewall, let the cluster auto‑heal. Monitor wsrep_cluster_status until it returns Primary and wsrep_ready = ON.
• Isolated node: restart mysqld on the node after network fix; it will request an SST or IST from a donor.
• Crashed node with gcache overflow: ensure a clean shutdown, then start mysqld. If the node’s gcache cannot hold recent writes, remove its data directory (e.g., rm -rf /var/lib/mysql/*) and let it rejoin to trigger a full SST.
• Misconfigured pc.ignore_sb: revert the option to its default (pc.ignore_sb=false) in wsrep_provider_options and restart the node; then verify that wsrep_cluster_status correctly reflects divergence.

When to Escalate

• More than 50 % of nodes report Non-Primary or Disconnected.
• Repeated SST failure after three attempts (check error log for SST failed).
• Suspected data divergence: run a checksum tool (e.g., pt-table-checksum) and confirm mismatch via wsrep_certify_status.

Limitations and Practical Verification

This guide detects split‑brain symptoms but cannot prove that divergent writes have not occurred without an explicit checksum. After recovery, run SHOW STATUS LIKE 'wsrep_cluster_status'; on all nodes and ensure each returns Primary and wsrep_ready = ON. Optionally, verify absence of split brain or gcache overflow messages in the error log.