Diagnosing RabbitMQ Publisher Confirm Failures That Cause Publishing Stalls

Recognizable condition

Publishers report increased latency or timeouts when sending messages. The broker log contains messages such as "channel error: precondition failed" or "no publishers confirm" for the affected connections.

Cause / diagnostic table

Possible cause	What to look for
Full queue or high memory usage	Queue length growing steadily; node memory approaching the high‑watermark.
Network partition preventing confirms from reaching the client	Dropped or retransmitted TCP packets; connection state shows "CLOSE_WAIT" or frequent reconnects.
Client library not handling confirm callbacks	Publish calls return immediately but no confirm/ack is processed; exceptions are swallowed.

Ordered checks

1. Verify queue length and memory usage

   Run on any RabbitMQ node (requires rabbitmqctl admin rights):

   rabbitmqctl list_queues name messages memory

   Look for queues with rapidly increasing messages or memory values approaching the node’s RAM limit.

   Risk: None; read‑only.

2. Check node health

   On the same node:

   rabbitmqctl status

   Examine the mem_limit and mem_used fields; also note any alarms (e.g., memory_alarm).

3. Inspect TCP connections for drops

   From the client host or the broker:

   netstat -anp | grep :5672

   or via RabbitMQ:

   rabbitmqctl list_connections name peer_host state

   Frequent transitions to closing or closed states indicate network instability.

4. Confirm client library handling

   Enable debug logging in the publisher (e.g., for Java client: com.rabbitmq.client level TRACE). Verify that:

   • The ConfirmListener.handleAck or handleNack callback is invoked for each publish.
   • Exceptions from the callback are not caught and ignored.

   If the library version is outdated, note it for upgrade.

Fixes tied to findings

• High queue length / memory pressure
  • Increase the queue length limit (if using a classic queue):

  rabbitmqctl set_policy HA-queue '^my.queue' '{"max-length":100000}' --apply-to queues
  

  Or add more consumers to drain the queue.

  • If memory is the bottleneck, raise the VM memory high watermark (requires node restart):

  # edit /etc/rabbitmq/rabbitmq.conf
  vm_memory_high_watermark = 0.7
  

  Then restart the node: systemctl restart rabbitmq-server.

  Rollback: Restore the previous vm_memory_high_watermark value and restart.

• Network partition
  • Restore connectivity between client and broker (check firewalls, VPN, load balancer).
  • Ensure mirrored queues are symmetric across nodes (ha-mode: all).

  After fixing, monitor rabbitmqctl list_connections for stable running state.

  Rollback: If you temporarily disabled a firewall rule, re‑enable it after verification.

• Client mis‑handles confirms
  • Upgrade to a recent client library version that properly exposes confirm callbacks.
  • Implement retry/back‑off logic on nack or timeout:

  // pseudo‑code
  if (!confirmReceivedWithin(timeout)) {
      backoffAndRetry();
  }
  

  Rollback: Revert to the previous library version if the new version introduces regressions.

Escalation criteria

• Publish latency remains > 2× baseline after applying the above fixes.
• Broker logs continue to show precondition failed errors despite healthy queues and connections.
• Memory alarms persist even after increasing vm_memory_high_watermark and adding consumers.

In these cases, consider:

• Opening a support ticket with RabbitMQ vendor, providing rabbitmqctl report output.
• Reviewing cluster topology for split‑brain scenarios.
• Conducting a controlled load test in a staging environment to isolate the issue.

Verification

After applying fixes:

1. Monitor publish latency via rabbitmq_exporter (Prometheus metric rabbitmq_queue_messages_publish_rate_total or latency histograms). Latency should return to pre‑incident baseline (e.g., <200 ms for 95th percentile).
2. Check broker logs for absence of channel error: precondition failed and no publishers confirm messages over a 10‑minute window.
3. Run a synthetic publisher that sends 1 000 messages with confirms enabled and asserts each publish receives a confirm within the configured timeout (<200 ms) for a sustained period (e.g., 5 min). No missed confirms should be reported.

Limitations

• Queue length checks may not detect transient spikes that resolve before the sampling interval.
• Network partition detection relies on TCP state; application‑level partitions (e.g., client‑side throttling) may not appear.
• Changing vm_memory_high_watermark requires a node restart, which briefly impacts availability.

Practical way to check the result: after any change, re‑run the ordered checks and confirm that the abnormal indicators (queue growth, memory usage, connection drops, missing confirm callbacks) are no longer present.