Diagnosing RabbitMQ HA Queue Failures: A Practical Checklist

Problem Statement

In a production RabbitMQ cluster, high‑availability (HA) queues sometimes fail to survive node outages or become unresponsive. The symptoms are:

• Consumers receive no messages after a node reboot.
• Queue status shows stopped or unavailable.
• Broker logs contain errors like queue replication failed or disk full.

These issues often stem from mis‑configured HA policies, incomplete quorum membership, or disk exhaustion on one or more nodes. A systematic diagnostic approach saves time and prevents accidental data loss.

Cause‑Diagnostic Table

Symptom	Likely Cause	Primary Check
HA queue disappears after node reboot	Cluster not fully formed or quorum not achieved	rabbitmqctl cluster_status
Queue status is stopped	HA policy misconfigured (wrong ha‑mode or ha‑params)	rabbitmqctl list_policies
Messages stop arriving despite consumers running	Disk full on a node hosting replicated queues	df -h or rabbitmqctl list_nodes disk_free
Logs show node marked non‑quorum member	Network partition or insufficient votes	rabbitmqctl cluster_status (quorum status)
Broker stalls on acknowledgments	Consumer or broker overload masking HA issues	Check consumer health and broker logs

Ordered Diagnostic Checklist

1. Verify Cluster Health
   

   Run on any node:

   rabbitmqctl cluster_status

   Look for:

   • All nodes listed under nodes
   • Quorum status shows quorum for each HA queue

   Risks: If a node is missing, the cluster is incomplete; avoid adding new nodes until the issue is resolved.

2. Check HA Queue Status
   

   rabbitmqctl list_queues name durable status

   Each HA queue should display running. If stopped or unavailable, proceed to policy verification.

3. Inspect HA Policy
   

   rabbitmqctl list_policies

   Confirm that the policy for the affected queues has:

   • ha-mode = exactly
   • ha-params = ["node1","node2","node3"] (exact node list)

   If ha-mode is nodes or exactly with incorrect nodes, the queue may not replicate correctly.

4. Validate Disk Availability
   

   On every node:

   df -h /var/lib/rabbitmq/mnesia

   Ensure at least 10 % free space. Alternatively, use:

   rabbitmqctl list_nodes disk_free

   Queues will be evicted if a node runs out of space.

5. Examine Broker Logs
   

   Open /var/log/rabbitmq/rabbit@<node>.log. Search for:

   • queue replication failed
   • disk full
   • node marked non‑quorum member

   These entries pinpoint the underlying issue.

6. Check Consumer Health
   

   Ensure consumers are running and acknowledging messages. A stalled consumer can appear as a queue failure.

Fixes Tied to Findings

• Cluster Incomplete or Quorum Lost
  

  Re‑join missing nodes:

  rabbitmqctl stop_app
  rabbitmqctl reset
  rabbitmqctl join_cluster rabbit@other_node
  rabbitmqctl start_app

  After re‑joining, verify cluster_status again.

• HA Policy Misconfiguration
  

  Update the policy (run on any node):

  rabbitmqctl set_policy ha_policy ".*" \'{\"ha-mode\":\"exactly\",\"ha-params\":\[\"rabbit@node1\",\"rabbit@node2\",\"rabbit@node3\"\]}\'\'\' durable\'

  Replace node names with your cluster’s hostnames. After setting, recreate the affected queues or let the policy apply to new ones.

• Disk Exhaustion
  

  Free space on the node:

  • Remove old log files in /var/log/rabbitmq
  • Delete unused queues with rabbitmqctl delete_queue (after confirming no pending messages)
  • Add additional disk space or move the Mnesia directory to a larger volume.

  After cleanup, restart the node.

• Node Marked Non‑Quorum Member
  

  Check network connectivity and firewall rules. Ensure all nodes can reach each other on ports 4369 and 25672. If a partition exists, resolve it before attempting to re‑join the quorum.

• Consumer or Broker Overload
  

  Scale consumers or adjust prefetch settings. Monitor broker memory and CPU usage with rabbitmqctl list_nodes and rabbitmqctl list_queues.

Escalation Criteria

• After completing all checks, if HA queues remain unavailable and logs show persistent replication errors.
• Disk space cannot be freed without risking data loss, and the cluster still fails to form quorum.
• Network partition persists beyond a reasonable time frame (e.g., >30 minutes).

At this point, contact RabbitMQ support or your infrastructure team to review the cluster topology, network configuration, and backup strategy.

Practical Verification Example

Assume a cluster with nodes rabbit@node1, rabbit@node2, rabbit@node3. The HA policy should replicate each queue to all three nodes.

rabbitmqctl list_policies
# Expected output snippet
ha_policy  ".*"  {"ha-mode":"exactly","ha-params":["rabbit@node1","rabbit@node2","rabbit@node3"]}  durable

Running rabbitmqctl list_queues name durable status should show:

my_queue  true  running

If the status is stopped, the queue is not active. Fix the policy or recreate the queue.

Limitations & Notes

• Changing HA policies on a live cluster can temporarily disrupt queue availability; perform changes during maintenance windows.
• Removing nodes from quorum without ensuring replication is complete may lead to data loss.
• Disk cleanup on a node hosting replicated queues may inadvertently delete unacknowledged messages; verify replication status first.

Always back up the Mnesia database before performing cluster resets or policy changes.