Deploying a Three‑Node MariaDB Galera Cluster

Problem Statement

Traditional asynchronous replication in MariaDB can leave a lag between a primary and its replicas, risking data loss during a failover. For applications that demand high availability and zero data loss, a Galera Cluster provides synchronous multi‑master replication: a transaction is committed only when all nodes agree.

Desired Outcome

Set up a three‑node Galera Cluster where each node can accept writes, each write is replicated synchronously, and the cluster recovers automatically from a single node failure without data loss. The cluster should expose a single logical endpoint to applications.

Prerequisites

• Three independent servers (VMs, bare metal, or containers) with identical OS (e.g., RHEL 9, Ubuntu 24.04). The OS must have the mysql user and group created before installing MariaDB.
• MariaDB 10.6 or newer and the matching Galera 4 library (the library package name and file path differ by distribution – e.g., /usr/lib64/galera-4/libgalera_smm.so on RHEL, /usr/lib/galera-4/libgalera_smm.so on Debian/Ubuntu).
• Network connectivity on the following ports, all open for both TCP and UDP: 3306 (MySQL client), 4567 (Galera replication), 4568 (Incremental State Transfer), 4444 (State Snapshot Transfer).
• SELinux or AppArmor configured to allow traffic on the ports above. For SELinux, run semanage port -a -t mysqld_port_t -p tcp 4567 4568 4444 and semanage port -a -t mysqld_port_t -p udp 4567 4568 4444.
• Identical sql_mode, default_storage_engine=InnoDB, and binlog_format=ROW on every node. Mismatched settings can cause replication errors.
• rsync installed on all nodes if you intend to use rsync as the SST method. If rsync is not available, choose mariabackup or xtrabackup instead.
• Optional but recommended: tune the write‑set cache size (wsrep_gcache_size) to at least the size of the largest transaction you expect to run. A small cache forces a full SST on node restart, which can be costly.

Configuration – my.cnf Example

Place the following in /etc/my.cnf.d/server.cnf (or the equivalent on your distribution). Replace placeholders with real values for each node.

[mariadb]
# General MySQL settings
binlog_format=ROW
default_storage_engine=InnoDB
innodb_autoinc_lock_mode=2

# Galera settings – common to all nodes
wsrep_on=ON
wsrep_provider=/usr/lib/galera-4/libgalera_smm.so   # adjust path for your distro
wsrep_cluster_name="production_cluster"
wsrep_cluster_address="gcomm://10.0.0.1,10.0.0.2,10.0.0.3"
wsrep_node_address="10.0.0.1"          # change per node
wsrep_node_name="node1"                 # change per node
wsrep_sst_method=rsync                    # or mariabackup/mariabackup
wsrep_gcache_size=128M                    # adjust to your workload
wsrep_sst_auth="user:password"          # if rsync requires SSH auth

# Optional tuning
wsrep_slave_threads=4
wsrep_local_state_timeout=120
wsrep_provider_options="gmcast.listen_addr=tcp://10.0.0.1:4567"   # optional, for debugging

Bootstrapping the First Node

Only the very first node that has no existing data should be bootstrapped. Running --wsrep-new-cluster on a node that already has data or is part of an existing cluster can create a split‑brain.

# Run on the first node (as root or sudo)
systemctl start mariadb --wsrep-new-cluster

After the command exits, the node will report wsrep_cluster_status=Primary and wsrep_local_state_comment=Synced. Verify with:

mysql -uroot -p -e "SHOW GLOBAL STATUS LIKE 'wsrep_cluster_size';"

Joining the Remaining Nodes

On the other two nodes, simply start MariaDB normally. They will contact the addresses listed in wsrep_cluster_address and perform a State Snapshot Transfer (SST) to synchronize data.

# On node2 and node3
systemctl start mariadb

During SST, the joining node will block until the transfer completes. Afterward, all nodes should show wsrep_cluster_size=3 and wsrep_cluster_status=Primary.

Cluster Health Checks

Run the following query on any node to confirm the cluster is healthy:

SHOW GLOBAL STATUS LIKE 'wsrep_%';

Variable	Expected Value	Meaning
wsrep_cluster_size	3	All nodes are present.
wsrep_cluster_status	Primary	Cluster is operational.
wsrep_ready	ON	Node will accept queries.
wsrep_local_state_comment	Synced	Node is fully up‑to‑date.
wsrep_flow_control_paused	0.0	No back‑pressure.

Practical test: insert a row on node1 and immediately query node2 and node3. The row should appear within milliseconds.

Failure Recovery

Node Crash or Restart

When a node is restarted, it will try an Incremental State Transfer (IST) using the write‑set cache (gcache). If the cache contains the missing transactions, IST completes quickly. If the cache is too small or the node was offline long enough that the cache has expired, a full SST will be triggered, which copies the entire dataset from a donor node.

Monitor the logs for messages such as INCREMENTAL STATE TRANSFER (IST) started or STATE SNAPSHOT TRANSFER (SST) started. A full SST can take minutes for large datasets.

Split‑Brain / Quorum Loss

If two nodes go down, the remaining node will lose quorum and block writes. To recover, stop the surviving node, then bootstrap it by running:

# On the surviving node only
systemctl stop mariadb
systemctl start mariadb --wsrep-new-cluster

After bootstrapping, start the two other nodes again. They will join via SST. This procedure should be used only when you are certain the majority of the original cluster is unavailable.

Consistent Backups

Use mariabackup with the --galera-info flag to capture a consistent snapshot of the cluster:

mariabackup --backup --target-dir=/backup/$(date +%Y%m%d%H%M%S) --galera-info

Restore the backup on a test server with mariabackup --prepare --target-dir=… and mariadb-restore to verify consistency.

Operational Limitations and Caveats

• Network Latency: Latency above ~2 ms between nodes can substantially reduce write throughput because each commit requires a round‑trip to all nodes. Place nodes in the same data center or availability zone when possible.
• Large Transactions: Transactions larger than the gcache or long‑running DDL can trigger flow control, pausing all writes until the transaction is replicated. Use tools like pt-online-schema-change for heavy schema changes.
• gcache Size: A too‑small wsrep_gcache_size forces full SST on node restart. Set it to at least the size of your largest expected transaction or the average size of data that can be generated while a node is offline.
• rsync Requirement: If you choose rsync as the SST method, ensure it is installed and that SSH keys or passwords are configured for wsrep_sst_auth. Without rsync, SST will fail unless another method is specified.
• Version Parity: All nodes must run the exact same MariaDB and Galera library versions. Mixing MariaDB 10.5 with Galera 4 from 10.6 can cause startup failures.
• Path Differences: The Galera provider library path differs between distributions. Verify the correct path on each node before bootstrapping.
• Bootstrapping Caution: The --wsrep-new-cluster flag should only be used on an empty node or during a controlled recovery. Using it on a node that already belongs to a cluster can split the cluster.

Summary

By following the steps above—installing matching MariaDB and Galera versions, configuring identical settings, bootstrapping one node, and joining the others—you can achieve a three‑node Galera Cluster that offers synchronous replication, automatic failover, and zero data loss in the event of a single node failure. Regular health checks, careful tuning of gcache, and awareness of network latency are essential for maintaining high performance.