Ensuring Consistent Restoration of Mirrored Queues with rabbitmq_snapshot

Question

Restoration Strategy & Verification
When a RabbitMQ cluster uses the rabbitmq_snapshot plugin to capture a point‑in‑time state, the goal is to restore that state without losing or duplicating messages, especially for mirrored queues that span multiple nodes.
Key constraints include the necessity of stopping the cluster (or taking each node offline) before restoration to avoid concurrent writes corrupting the replayed state.  Restoring only a subset of nodes can leave the cluster in a split‑brain condition, causing inconsistent replication status and potential message loss.
Verification typically relies on comparing pre‑restore and post‑restore queue depths, consumer acknowledgment counts, and message payload checksums.  However, the ordering of unacknowledged messages is not guaranteed to be preserved, which can affect deterministic consumers.
Unresolved decision points remain about the optimal restoration sequence for mirrored queues and the most reliable automated verification approach.
Which restoration sequence guarantees that mirrored queues remain consistent across all nodes?  How can we programmatically detect duplicate or missing messages after a snapshot restore?  What verification metrics should be collected to confirm the ordering of acknowledged versus unacknowledged messages?

Tasadduq Burney · Answer

Consistent Restoration Sequence
To guarantee that mirrored queues remain consistent across all nodes, you must perform a full cluster shutdown and a simultaneous state reset. Restoring a subset of nodes or restoring nodes while others are active creates a split-brain scenario where Mnesia metadata diverges, leading to replication failures or message loss.

Required Restoration Steps

Stop All Nodes: Shut down every RabbitMQ node in the cluster to prevent concurrent writes or partition events during the restore.
Clear Mnesia Data: Manually delete the existing Mnesia database directories on all nodes. This ensures no stale metadata or partial mirror states persist.
Apply Snapshot: Deploy the rabbitmq_snapshot to all nodes.
Synchronized Startup: Restart the nodes. The cluster will rebuild the mirror topology based on the restored global state provided by the snapshot.

Programmatic Detection of Data Loss or Duplication
Because rabbitmq_snapshot restores the internal state of the broker, it cannot natively identify messages that were produced after the snapshot was taken but before the restoration. To detect these gaps or duplicates programmatically:

External Reconciliation: Compare the messages_ready and messages_unacknowledged counts via the Management API against an external source of truth (e.g., a transactional database or a producer-side audit log).
Payload Checksumming: For critical data, sample message payloads post-restore and compare their checksums against the known state of the data source at the time of the snapshot.

Verification of Message Ordering
It is important to note that strict ordering of unacknowledged messages is not guaranteed during a snapshot recovery. To verify the state of acknowledged versus unacknowledged messages, collect the following metrics via rabbitmq-diagnostics or the API:

Metric
Verification Purpose

messages_ready
Confirms the volume of messages awaiting delivery matches the snapshot point.

messages_unacknowledged
Identifies messages that were in-flight during the snapshot; these may be redelivered out of order.

cluster_status
Ensures all mirrored queues have synchronized their state across the restored nodes.

Verification Command:
rabbitmq-diagnostics cluster_status

Assumptions and Uncertainty
This guidance assumes the use of RabbitMQ 3.x. Behavior may vary in version 4.x or newer regarding Mnesia compatibility. We assume the snapshot was taken while the cluster was in a healthy state; snapshots taken during a partition may propagate that inconsistency upon restoration.

Missing Diagnostic Detail: Are you utilizing a specific version of the rabbitmq_snapshot plugin or a custom implementation? The specific version may alter the required Mnesia cleanup procedure.

Ensuring Consistent Restoration of Mirrored Queues with rabbitmq_snapshot

Restoration Strategy & Verification

1 answer

Consistent Restoration Sequence

Required Restoration Steps

Programmatic Detection of Data Loss or Duplication

Verification of Message Ordering

Assumptions and Uncertainty

Your perspective belongs here.

0 question comments

Metric	Verification Purpose
`messages_ready`	Confirms the volume of messages awaiting delivery matches the snapshot point.
`messages_unacknowledged`	Identifies messages that were in-flight during the snapshot; these may be redelivered out of order.
`cluster_status`	Ensures all mirrored queues have synchronized their state across the restored nodes.