Scaling Message Workers with NATS JetStream Pull Consumers

Problem: Scaling Workers Without a Load Balancer

In many event‑driven architectures you have a stream of domain events (orders, payments, logs) that must be processed by a fleet of worker instances. The naive approach is to let the broker push messages to each worker, but that ties the server’s delivery rate to the slowest consumer and requires a separate load‑balancing layer to split the work.

JetStream’s pull consumers solve this by letting each worker pull a batch of messages it can handle, giving the worker full control over throughput while still maintaining at‑least‑once delivery. The broker only sends what the worker requests, and if a worker crashes its unacknowledged messages are automatically redelivered to another instance in the same consumer group.

How Pull Consumers Work

A pull consumer is a durable view over a stream. It keeps a cursor (the last delivered message sequence) and exposes two main operations:

• Fetch(batchSize) – the client requests a batch of messages. The broker returns up to batchSize messages that have not yet been acknowledged.
• Ack(message) – after processing a message the client sends an acknowledgement. If the ack is not received before the consumer’s AckWait timeout, the message is redelivered.

When multiple consumers share the same durable name, they form a queue group. Each message is delivered to only one consumer in the group, so adding a new worker automatically increases throughput. No external load balancer is required.

Setting Up a Pull Consumer

Below is a minimal example using the nats CLI and a Go client. The same concepts apply to other official clients.

# Start a single‑node JetStream server
nats-server -js

# Create a stream that stores order events
nats stream add ORDERS \
  --subjects orders.* \
  --storage file \
  --replicas 1

# Create a durable pull consumer named "order-workers"
nats consumer add ORDERS order-workers \
  --durable \
  --deliver-policy all \
  --ack-policy explicit \
  --ack-wait 30s \
  --max-deliver 5

In the Go client you would set up the consumer as follows:

js, err := nc.JetStream()
if err != nil {
    log.Fatal(err)
}

// Create or retrieve the durable pull consumer
consumer, err := js.PullSubscribe("orders.*", "order-workers", nats.Bind("ORDERS"))
if err != nil {
    log.Fatal(err)
}

Now each worker can repeatedly call consumer.Fetch(10) to pull up to ten messages at a time.

Worker Loop Example

for {
    msgs, err := consumer.Fetch(10, nats.MaxWait(5*time.Second))
    if err != nil {
        if err == nats.ErrTimeout {
            continue // no messages available yet
        }
        log.Fatal(err)
    }
    for _, msg := range msgs {
        // Process the order
        if err := handleOrder(msg.Data); err != nil {
            // Do not ack – will be redelivered
            continue
        }
        // Ack on success
        msg.Ack()
    }
}

Because the consumer is durable, the broker remembers the cursor across worker restarts. If a worker crashes before acking a message, the message will be redelivered to another worker in the group once the AckWait expires.

Trade‑offs & Limitations

• At‑least‑once delivery means you may see duplicates. Your handler must be idempotent or deduplicate using the message Msg.ID field.
• AckWait tuning is critical. A too‑short timeout can cause spurious redeliveries if a worker is busy; a too‑long timeout delays recovery from crashes.
• Batch size vs latency – larger batches reduce round‑trip overhead but increase memory usage and can delay the first message in a batch if the worker is slow.
• Storage type – file storage gives durability across restarts but is slower than memory storage; choose based on your retention and throughput needs.
• Cluster vs single node – a clustered JetStream setup with replicas improves availability, but adds operational complexity and requires careful configuration of replication factors and storage limits.

Verification Checklist

1. Confirm the stream exists: nats stream info ORDERS.
2. Check the consumer’s state: nats consumer info ORDERS order-workers. Look for Delivered and Pending counts.
3. Publish a test message: nats pub orders.new '{"id":1,"item":"widget"}'.
4. Run two worker instances and observe that each receives a distinct message from the same stream.
5. Kill one worker mid‑processing and verify that its unacknowledged message is redelivered to the remaining worker after AckWait.
6. Inspect the MaxDeliver setting to ensure messages aren’t retried indefinitely.

Actionable Takeaway

Pull consumers are the idiomatic way to horizontally scale NATS JetStream workers when you need batch processing, flow control, and at‑least‑once semantics. By creating a durable pull consumer and launching multiple worker instances that fetch messages in batches, you eliminate the need for external load balancing and gain built‑in fault tolerance. Just remember to make your handlers idempotent, tune AckWait and MaxDeliver for your workload, and verify the consumer state with the CLI before deploying to production.