Diagnosing Apache Pulsar Consumer Ack Failures: A Practical Checklist

Problem Statement

When a Pulsar consumer misbehaves—duplicating messages, losing them, or stalling—most root causes lie in the acknowledgment (ack) configuration. This guide walks through the most common ack‑mode pitfalls and provides a step‑by‑step diagnostic flow to pinpoint and resolve them.

Recognizable Symptoms

• High duplicate delivery rate (≥ 5% of messages)
• Messages never appear in the application after a crash or pause
• Consumer stalls with a growing backlog of unacknowledged messages
• Unexpected re‑delivery of processed messages on restart

Cause‑Diagnostic‑Fix Table

Cause	Diagnostic Check	Fix
autoAck=true while business logic requires manual ack	Check consumer config for autoAck=true	Set autoAck=false and implement consumer.acknowledge(msg) after processing.
ackTimeoutMs set too low (< 1000 ms)	Run pulsar-admin consumers list <topic> and look for ackTimeoutMs value.	Increase timeout to at least 2× average processing latency.
BatchAcknowledgment used without ensuring all batch messages belong to the same transaction	Inspect code path that calls consumer.acknowledgeBatch and verify idempotency.	Switch to single‑message ack or guarantee batch idempotence.
ackTimeoutMs set to 0 (no timeout)	Check broker logs for ACK_TIMEOUT=0 entries.	Set a non‑zero timeout or enable negativeAck if supported.
Shared subscription with manual ack and unsynchronized ack cursors	Verify that each consumer thread tracks its own cursor via consumer.acknowledge.	Use Shared only with autoAck or move to Failover if ordering is critical.
NegativeAck ignored due to unsupported broker version	Check broker version; older BookKeeper releases lack negative ack support.	Upgrade broker or avoid consumer.negativeAck.

Diagnostic Flow

1. Confirm the Consumer Configuration
   

   In Java:

   Consumer<String> consumer = client.newConsumer(Schema.STRING)
       .topic("my-topic")
       .subscriptionName("my-sub")
       .subscriptionType(SubscriptionType.Shared)
       .ackTimeout(5000, TimeUnit.MILLISECONDS) // 5 s
       .autoAck(false)
       .subscribe();
   

   Run the Pulsar CLI to double‑check:

   pulsar-client consume my-topic --subscription-name my-sub --ack-timeout-ms 5000

   • Ensure autoAck matches application logic.
   • Verify ackTimeoutMs is reasonable.
2. Inspect Broker State
   

   Use the admin API:

   pulsar-admin consumers list my-topic

   Look for:

   • unackedMessages count.
   • Current ackCursor position.
   • Any pendingAck entries.
3. Check Logs for Timeout or Batch Warnings
   

   Search broker logs for patterns:

   grep -i "ACK_TIMEOUT" /var/log/pulsar/pulsar-logs/*
   grep -i "BATCH_ACK" /var/log/pulsar/pulsar-logs/*

   These entries indicate brokers enforcing the configured timeout or batch semantics.

4. Simulate a Crash Scenario
   

   1. Start the consumer and let it receive a message. 2. Terminate the process (Ctrl‑C or kill).

   3. Restart the consumer and observe if the message is redelivered after the ack timeout period.

   Use pulsar-admin consumers list to confirm the message re‑enters the unacked queue.

5. Validate Ack Cursor Advancement
   

   After a successful ack, run:

   pulsar-admin consumers list my-topic | grep ackCursor

   The cursor should move forward. If it does not, the ack may be ignored.

6. Escalation Criteria
   
   • Duplicate rate > 5% over a 5‑minute window.
   • No redelivery after a controlled crash for > ackTimeoutMs.
   • Unacked messages > 10 % of total messages for > 30 minutes.

   When any threshold is crossed, open a support ticket with Pulsar community or vendor, providing:

   • Consumer config snapshot.
   • Broker log excerpts.
   • Metrics from pulsar-admin consumers list.

Practical Checkpoints

• Run pulsar-admin topics stats my-topic to see UnackedMessages trend.
• Use Prometheus metrics pulsar_consumer_acked_messages_total and pulsar_consumer_unacked_messages to monitor ack health.
• Enable broker ackTimeoutMs alerts in your monitoring stack.

Limitations

• These diagnostics assume Pulsar 2.10+; older versions may lack certain CLI flags.
• Batch ack behavior depends on broker's BookKeeper version; ensure compatibility.
• NegativeAck support is broker‑only; if unsupported, the client will silently ignore the call.

Final Takeaway

Accurate ack configuration is the linchpin of reliable Pulsar consumption. By systematically validating autoAck, ackTimeoutMs, batch usage, and subscription type, you can eliminate duplicate or lost messages and keep your consumer fleet healthy.