Diagnosing Persistent Flow Control in MariaDB Galera Clusters

The Recognizable Condition

Your Galera Cluster application latency spikes periodically, and when you check the wsrep status variables, wsrep_flow_control_paused or wsrep_flow_control_sent keep climbing. Flow control is Galera's built-in back-pressure mechanism: when one node cannot apply replicated writes fast enough, the cluster pauses the fast nodes so the slow one can catch up. Occasional activation is normal; sustained activation means something is genuinely wrong.

Important: these counters are cumulative since server start. A single reading tells you almost nothing. Take two samples a known interval apart (e.g., 60 seconds) and compute the rate of change before drawing conclusions.

Before You Diagnose Flow Control

Flow control counters are only meaningful on a healthy, joined cluster. Confirm first:

SHOW GLOBAL STATUS LIKE 'wsrep_cluster_status';
SHOW GLOBAL STATUS LIKE 'wsrep_ready';
SHOW GLOBAL STATUS LIKE 'wsrep_cluster_size';
SHOW GLOBAL STATUS LIKE 'wsrep_local_state_comment';

You want wsrep_cluster_status = Primary, wsrep_ready = ON, the expected cluster size, and wsrep_local_state_comment = Synced. If a node is in Donor/Desynced or Joined state, flow control pauses may simply reflect state transfer, not a capacity problem.

Cause and Diagnostic Table

Possible Cause	Typical Symptoms	Key Diagnostics
Slow apply node (CPU, I/O, or memory saturation)	One node lags; wsrep_flow_control_paused rises on others	top, iostat, vmstat on the lagging node; compare wsrep_local_recv_queue across nodes
Network latency or packet loss between nodes	Cluster-wide pauses, no single slow node	ping/mtr between nodes; check for retransmits in netstat -s
Write bursts exceeding apply capacity	Counters spike during batch jobs or imports	Correlate counter spikes with workload schedule; check wsrep_replicated_bytes rate
Flow control trigger set too low	Pauses even under modest load	Inspect gcs.fc_limit in wsrep_provider_options; compare against wsrep_local_recv_queue peaks
Large transactions	Sharp pauses tied to specific statements	Check wsrep_max_ws_size, binary log / slow query log for huge transactions

Ordered Checks

1. Sample the flow control counters over time. On each node, run twice, 60 seconds apart:

   SHOW GLOBAL STATUS LIKE 'wsrep_flow_control_paused';
   SHOW GLOBAL STATUS LIKE 'wsrep_flow_control_sent';
   SHOW GLOBAL STATUS LIKE 'wsrep_local_recv_queue';
   

   Compute the delta. wsrep_flow_control_paused is a fraction of time paused, so a delta of 0.05 over 60 seconds means the node spent roughly 3 seconds paused. A queue that keeps growing points to a slow applier.
2. Identify which node is the bottleneck. Compare wsrep_local_recv_queue and wsrep_local_send_queue across all nodes at the same moment. The node with the persistently largest receive queue is usually the one triggering flow control.
3. Check resources on the lagging node. Run from the OS shell (no special privileges needed for read-only inspection):

   top -b -n1 | head -20
   iostat -dx 1 5
   vmstat 1 5
   

   Look for sustained CPU above ~80%, high iowait, or swap activity. Disk I/O saturation on the applier is one of the most common causes.
4. Check the network path. From one node to another:

   ping -c 20 
   mtr --report 
   

   Any packet loss, or RTT far above your baseline, warrants network investigation before touching Galera settings.
5. Review the provider options actually in effect.

   SHOW GLOBAL VARIABLES LIKE 'wsrep_provider_options';
   

   Note the values of gcs.fc_limit (receive-queue depth that triggers flow control), gcs.fc_factor (resume threshold as a fraction of the limit), and gcs.fc_master_slave. Defaults vary by Galera library version — check your installed version's documentation rather than assuming values.
6. Correlate with workload. Check the slow query log or Performance Schema for write bursts, large transactions, or DDL (which in older Galera versions uses Total Order Isolation and stalls the cluster).

Fixes Tied to Findings

• Slow apply node: Relieve the bottleneck — faster storage for the data directory, more RAM for the InnoDB buffer pool, or move read traffic off that node. If the node is simply undersized relative to the write load, resize it. Verify afterward that its receive queue drains under peak load.
• Network issues: Fix the path (cabling, switch configuration, routing). If you consider jumbo frames, every device on the path — NICs, switches, routers — must support the larger MTU, or you will get fragmentation and worse performance. Re-test with ping/mtr after changes.
• Flow control triggering too early: You can raise gcs.fc_limit so a deeper queue is tolerated before pausing. This changes state, so plan a rollback: record the current full wsrep_provider_options string first, then append or modify the option while preserving all existing values:

  SET GLOBAL wsrep_provider_options = 'gcs.fc_limit=256; ...existing options...';
  

  Note: wsrep_provider_options is a single string — setting it replaces the whole string, so omitting existing options resets them to defaults. To roll back, restore the original string the same way. Test in staging: raising the limit only delays flow control; if the applier genuinely cannot keep up, queues will grow until memory pressure becomes the new problem.
• Write bursts / large transactions: Break bulk operations into smaller transactions, stagger batch jobs, or throttle the writer at the application level. This is usually safer than tuning thresholds.
• Cluster-wide capacity exceeded: Adding nodes to a Galera cluster adds read capacity but does not reduce per-node write apply load — every node applies every write. If writes are the bottleneck, consider sharding or offloading read traffic so appliers keep up. Never drop below three nodes (or add an arbitrator) to preserve quorum.

Verifying the Fix

Log the counters periodically without embedding credentials on the command line. Create a defaults file readable only by root:

# /root/.my.cnf  (chmod 600)
[client]
user=monitor
password=<your-password>

Then a cron entry (run as root on each node) can sample safely:

* * * * * /usr/bin/mysql --defaults-file=/root/.my.cnf -e "SHOW GLOBAL STATUS LIKE 'wsrep_flow_control_paused';" >> /var/log/galera_flow.log 2>&1

Track the rate of change, not raw values. Success looks like the paused fraction staying near zero through your normal peak load, with application latency back within expectations.

When to Escalate

• The paused-time fraction (computed from counter deltas, not raw values) stays above a level that breaches your latency SLA — for example, more than 10% of a 15-minute window — after node-level fixes.
• Multiple nodes pause simultaneously and no single bottleneck node can be identified, suggesting a cluster-wide or network-wide cause.
• You have verified the write workload simply exceeds what any single node can apply — an architectural limit of synchronous replication, not a tuning problem.

Escalation means an architecture review: workload distribution, sharding, or whether synchronous multi-primary replication is the right fit for this workload.

Limitations and Cautions

• Raising flow control thresholds masks symptoms. If the underlying cause is a slow disk or bad network, you are trading brief pauses for unbounded queue growth.
• Exact option names, defaults, and semantics (gcs.fc_limit, gcs.fc_factor, gcs.fc_master_slave) depend on the Galera library version bundled with your MariaDB release. Confirm against the documentation for your version before changing anything.
• Some provider option changes require a node restart to take effect; verify with SHOW GLOBAL VARIABLES LIKE 'wsrep_provider_options'; after any change.