RabbitMQ Message Durability Across Broker Restarts

Message Loss After a Broker Restart

If a RabbitMQ broker stops and starts, any queue or message that was not explicitly marked for durability disappears. Non-durable queues and transient messages exist only in memory. For workloads where a lost message means a failed order, a missed alert, or an incomplete state change, the application must enforce a durability contract at the publisher and consumer level.

The useful takeaway is narrow: declare the queue as durable and set every critical message to delivery mode 2. Neither setting alone is sufficient.

The Two Required Settings

RabbitMQ persists a message across restarts only when both sides of the contract are satisfied.

• Durable queue: The queue definition is written to disk when the queue is declared. If the queue is not durable, the broker forgets it exists after a restart, even if messages were persistent.
• Persistent message: The message property delivery_mode must be 2. This tells the broker to write the message body to disk before sending a publish confirmation. Transient messages use delivery mode 1 and remain in memory only.

Durability is a broker-local guarantee. It protects against RabbitMQ process restarts and host reboots, but not against disk failure or data center loss.

Smallest Suitable Design

For a single-node RabbitMQ deployment handling critical asynchronous work, the minimal design is:

1. Declare all application queues with durable=True.
2. Publish all business-critical messages with delivery_mode=2.
3. Use manual consumer acknowledgments so unacknowledged messages are redelivered if a consumer crashes.
4. Enable publisher confirms so the producer knows the broker accepted the message.

Non-durable queues and transient messages are acceptable only for disposable data such as telemetry or retryable logs where loss is cheaper than the disk I/O cost of durability.

Trust and Data Boundaries

The publisher trusts the broker to acknowledge receipt only after the message is safely queued. The broker trusts the local filesystem to complete write operations. The consumer trusts the broker to redeliver unacknowledged messages after a failure. Each boundary can fail independently:

Configuration Example

The following pseudocode shows the AMQP semantics. It is not a tested program; adapt it to your client library.

# Declare a durable queue
channel.queue_declare(queue='order_events', durable=True)

# Publish a persistent message
channel.basic_publish(
    exchange='',
    routing_key='order_events',
    body=b'{"order_id": "12345"}',
    properties={'delivery_mode': 2}
)

If the queue was previously declared as non-durable, redeclaring it as durable returns an error. Delete or rename the existing queue first, or use a different queue name.

Operational Verification

Verify durability on a staging broker before relying on it in production:

1. Declare a durable queue named verify_durable.
2. Publish one persistent message to that queue.
3. Stop the RabbitMQ service completely.
4. Restart the service.
5. Query the queue with rabbitmqctl list_queues name durable messages or open the management UI.
6. Confirm the message count is still one and consume it to confirm payload integrity.

Run rabbitmqctl on the broker host as a user with administrator privileges, typically the rabbitmq system user or through sudo. Restarting the broker drops all client connections, so schedule this during a maintenance window.

Failure Modes and Design Changes

• Slow disk: Persistent messages force disk synchronization. High publish latency or publisher timeouts usually indicate disk I/O saturation. Monitor disk write latency, not just queue depth.
• Disk full: RabbitMQ blocks publishers when the disk free space falls below the configured limit. Durable queues do not prevent this; they make it more likely under load.
• Broker cluster partition: A durable queue on one node is not automatically available on another. Use mirrored or quorum queues if node failure must be tolerated.
• Consumer crashes: Durability does not replace acknowledgments. Without manual acks, RabbitMQ deletes the message upon delivery, and a consumer crash loses it.

Change the design if you need survival beyond a single host restart. Add RabbitMQ clustering with quorum queues for node failure, or replicate to a secondary site for data center failure. Those designs add operational complexity and cross-network latency.

Limitations and Practical Check

Durability guarantees storage on the broker's local disk only. It is not a backup, and it does not protect against logical errors such as publishing to the wrong exchange. A practical production check is to graph disk write latency alongside ready and unacknowledged message counts. If write latency rises while queue depth stays flat, the durability path is likely the bottleneck.