Blue-green queue migration in RabbitMQ: drain in place or shovel to the new topology

Question

We need to change immutable queue arguments (for example x-queue-type and max-length) on a set of classic queues in production. Since redeclaring with different arguments raises 406 PRECONDITION_FAILED, the plan is a blue-green approach: declare new queues and bindings alongside the old ones, move consumers over, then remove the old topology.

The unresolved decision is how to handle messages still sitting in the old queues at cutover time. One option is to keep consumers bound to both old and new queues until the old ones drain to zero. The other is to use the shovel plugin to move remaining messages into the new queues, accepting that ordering across the two topologies is not preserved and that duplicates are possible if acks are lost mid-transfer.

Messages are roughly order-sensitive per queue, and some consumers are not fully idempotent today. We are on a recent 3.x release and can verify behavior in a staging cluster first.

Given non-idempotent consumers, is dual-binding consumers during a drain generally safer than shoveling, or does the dual-binding window introduce its own ordering hazards?
Does shoveling preserve per-message ordering well enough in practice for a single source/target queue pair, or should ordering be assumed broken?
Is there a documented signal (beyond queue depth metrics) for confirming the old queue is fully drained before deleting it?

Tasadduq Burney · Answer

Recommended Migration Strategy
For non-idempotent consumers and order-sensitive messages, dual-binding consumers (drain-in-place) is generally safer than shoveling, provided you accept a temporary window of out-of-order processing across the two topologies.

Comparison of Approaches
The primary risk in your scenario is the lack of consumer idempotency. Here is how the two strategies impact your constraints:

Factor
      Dual-Binding (Drain)
      Shoveling

Ordering
      Preserved per-queue; broken across topologies.
      Preserved per-message; broken across topologies.

Duplicates
      Low (standard RabbitMQ ack risks).
      Higher (risk of duplication during transfer/ack failure).

Complexity
      Low (consumer config change).
      Medium (plugin config and monitoring).

Detailed Analysis

1. Dual-Binding Hazards
Dual-binding does not introduce new ordering hazards within a single queue, but it does mean a message produced to the new queue might be processed before a message produced to the old queue. If your business logic requires strict global ordering across the migration boundary, neither method is sufficient without a sequencing key at the application level. However, it avoids the double-handling and potential duplication inherent in moving messages between queues.

2. Shoveling and Ordering
The Shovel plugin preserves the order of messages as they are read from the source and written to the destination. In practice, for a single source-to-target pair, the sequence is maintained. However, because you are moving messages into a queue that is already receiving live traffic from producers, the "shoveled" messages will appear at the end of the new queue, effectively jumping behind any messages already produced to the new topology.

3. Verification of Drain
There is no single "drained" event signal. You must rely on the messages and messages_ready metrics. A queue is considered fully drained when:

messages count is 0.
  messages_unacknowledged is 0 (ensuring no consumer is still holding a message).

Implementation Steps for Drain-in-Place

Declare the new "Green" queues with the updated arguments.
  Update bindings so producers send to both or switch producers to the Green exchange/queue.
  Configure consumers to listen to both the Blue and Green queues simultaneously.
  Monitor the Blue queue depth via the Management API or CLI: rabbitmqctl list_queues name messages messages_ready
  Once Blue messages and messages_unacknowledged reach 0, remove the Blue queue and its bindings.

Missing Diagnostic: Are your consumers using a single-threaded prefetch (basic.qos prefetch_count=1)? If prefetch is high, the "drain" phase may take longer as messages remain unacknowledged in consumer buffers despite the queue appearing empty.

Blue-green queue migration in RabbitMQ: drain in place or shovel to the new topology

1 answer

Recommended Migration Strategy

Comparison of Approaches

Detailed Analysis

1. Dual-Binding Hazards

2. Shoveling and Ordering

3. Verification of Drain

Implementation Steps for Drain-in-Place

Your perspective belongs here.

0 question comments

Factor	Dual-Binding (Drain)	Shoveling
Ordering	Preserved per-queue; broken across topologies.	Preserved per-message; broken across topologies.
Duplicates	Low (standard RabbitMQ ack risks).	Higher (risk of duplication during transfer/ack failure).
Complexity	Low (consumer config change).	Medium (plugin config and monitoring).