Blue-green queue migration in RabbitMQ: drain in place or shovel to the new topology
19.8K reputation · 26 Jan 2024, 09:25 UTC
We need to change immutable queue arguments (for example x-queue-type and max-length) on a set of classic queues in production. Since redeclaring with different arguments raises 406 PRECONDITION_FAILED, the plan is a blue-green approach: declare new queues and bindings alongside the old ones, move consumers over, then remove the old topology.
The unresolved decision is how to handle messages still sitting in the old queues at cutover time. One option is to keep consumers bound to both old and new queues until the old ones drain to zero. The other is to use the shovel plugin to move remaining messages into the new queues, accepting that ordering across the two topologies is not preserved and that duplicates are possible if acks are lost mid-transfer.
Messages are roughly order-sensitive per queue, and some consumers are not fully idempotent today. We are on a recent 3.x release and can verify behavior in a staging cluster first.
- Given non-idempotent consumers, is dual-binding consumers during a drain generally safer than shoveling, or does the dual-binding window introduce its own ordering hazards?
- Does shoveling preserve per-message ordering well enough in practice for a single source/target queue pair, or should ordering be assumed broken?
- Is there a documented signal (beyond queue depth metrics) for confirming the old queue is fully drained before deleting it?