Build at‑least‑once delivery in RabbitMQ: publisher confirms + manual acks

Why you can’t trust a successful publish call

When a producer calls basic.publish, the broker accepts the message on the channel, but that does not mean the message reached a queue. If the exchange has no matching binding, RabbitMQ can drop the message silently. The application sees no error and assumes the message is safe.

The key takeaway is that a producer must ask the broker to confirm acceptance, and a consumer must explicitly acknowledge that it has finished processing. Together they give you at‑least‑once delivery – the broker guarantees the message was stored, and the consumer guarantees the work was done.

Publisher confirms: the broker’s “I got it” signal

Publisher confirms are a RabbitMQ extension. When a channel runs confirm.select, the broker replies with basic.ack after it has written the message to a queue, or basic.nack if it cannot route the message (e.g. no binding and mandatory flag set). The producer can then decide whether to retry, log, or route the message to a dead‑letter exchange.

In the Python pika client this looks like:

channel.confirm_select()
channel.basic_publish(exchange='orders', routing_key='new', body='{}', properties=pika.BasicProperties(delivery_mode=2))
# Wait for the broker’s ack
channel.wait_for_confirms()  # raises exception if any message was nacked

Individual confirms give you a 1:1 mapping between a message and its ack, which is great for debugging. For high‑throughput producers you can enable confirm_select and use channel.wait_for_confirms(timeout=0.5) or channel.add_on_return_callback to batch the checks – this reduces round‑trip traffic but makes it harder to pinpoint the exact failed message.

Manual consumer acknowledgments: keeping work safe

Most client libraries auto‑acknowledge messages as soon as they are delivered. If a consumer crashes after receiving a message but before finishing its work, the message is lost forever. With basic.qos(prefetch_count=1) and basic_ack you keep the message in an unacknowledged state until you explicitly tell RabbitMQ you’re done.

Example in Python:

def on_message(channel, method, properties, body):
    try:
        # Persist the payload first – this is the idempotent step
        save_to_db(body)
    except Exception:
        channel.basic_nack(delivery_tag=method.delivery_tag, requeue=False)
        raise
    else:
        channel.basic_ack(delivery_tag=method.delivery_tag)

channel.basic_consume(queue='orders', on_message_callback=on_message, auto_ack=False)
channel.start_consuming()

If the consumer dies before calling basic_ack, RabbitMQ will redeliver the message to the same or another consumer. That’s why duplicates are expected; the consumer must be idempotent or perform deduplication.

Concrete pattern: confirm, persist, acknowledge

Below is a step‑by‑step recipe you can copy into a production system. Replace placeholders like <QUEUE> and <EXCHANGE> with your own names.

1. Producer side – open a channel, enable confirms, and publish with mandatory=True so you get a basic.nack if the message cannot be routed.

   channel.confirm_select()
   channel.basic_publish(exchange='orders', routing_key='new', body='{}', mandatory=True, properties=pika.BasicProperties(delivery_mode=2))
   channel.wait_for_confirms()  # blocks until broker replies
   

2. Consumer side – set prefetch to 1, consume with auto_ack=False, write to durable storage, then acknowledge.

   channel.basic_qos(prefetch_count=1)
   channel.basic_consume(queue='orders', on_message_callback=on_message, auto_ack=False)
   channel.start_consuming()
   

3. Idempotency check – store a unique message ID (e.g. a UUID or the broker’s delivery_tag) in a database before the write. On a duplicate, the write will be a no‑op or a safe update.

Verifying the queue type

For replicated queues you should use quorum queues in RabbitMQ 3.10+. Classic mirrored queues are still available but deprecated. You can check the queue type with rabbitmqctl or via the Management UI.

# Run on a node where you have rabbitmqctl access
rabbitmqctl list_queues name type

Look for quorum in the type column. If you see classic and the cluster is running 3.10+, consider migrating to a quorum queue.

Testing redelivery

1. Start a consumer with manual acks. 2. Let it fetch a message but stop the process before calling basic_ack. 3. Close the connection (Ctrl‑C or kill the process). 4. Observe that the same message appears in the queue again – either to the same consumer or to a new one.

Use logging to capture the message ID and timestamp. When the consumer restarts, it should process the duplicate without corrupting state.

Trade‑offs and limitations

• Confirm + manual ack gives at‑least‑once delivery, not exactly‑once. Duplicates are inevitable when a consumer crashes or a network partition occurs.
• Publisher confirms add latency: the producer blocks until the broker replies. Measure latency on your workload; if you need sub‑millisecond throughput, consider batching confirms or using a separate monitoring channel.
• Queue TTL, max‑length eviction, or basic.nack with requeue=false can still discard messages, even if you use confirms and acks. Plan for dead‑letter exchanges if you need to capture such failures.
• Classic mirrored queues are deprecated in RabbitMQ 3.10+. If you are on an older release, they still work, but migrating to quorum queues is recommended for future‑proofing.

Actionable next steps

1. Enable confirm_select on all production producers. 2. Switch consumers to auto_ack=False and add a persistence step before basic_ack. 3. Verify queue replication type with rabbitmqctl list_queues name type and migrate classic mirrored queues to quorum queues if you are on 3.10+. 4. Run a smoke test: kill a consumer mid‑processing and confirm the message is redelivered. 5. Monitor confirm latency and ack throughput; adjust prefetch or batch size as needed.

With these changes you will see explicit errors when a message cannot be routed, and you will guarantee that every message that reaches a consumer is processed at least once. Duplicates are a reality, but the pattern above gives you the tools to handle them cleanly.