NATS JetStream: Choosing a Retention Policy for a Durable Event Backbone

Requirements

The backbone must deliver ordered, replayable events with at-least-once semantics to multiple independent consumer groups. Retention must survive broker restarts and allow late joiners to catch up without missing events within the retention window. Workloads include order processing, audit logging, and cross-service state synchronization. The system must tolerate broker crashes, network partitions, and consumer restarts.

Smallest suitable design

Begin with a single JetStream stream using the subject wildcard events.>. This captures all event subjects under one namespace, simplifying ACLs and monitoring. Set storage=file so messages persist across restarts. Configure max_bytes to your provisioned disk budget; keep max_msgs=-1 so message count isn't a hard limit. Choose a discard policy that matches your backpressure preference: discard=new rejects publishes when the byte cap is hit, backpressuring producers; discard=old evicts the oldest messages first, keeping the most recent window; interest retains only what at least one consumer still needs, auto-cleaning but blocking replays for new groups. With three replicas (replicas=3) in a clustered deployment, writes require a 2/3 quorum. Add one durable pull consumer per consumer group with ack_wait tuned to your handler's runtime (the default 30s is often too short), max-deliver set to 5, and a dead-letter subject configured to capture poison messages instead of stalling the consumer.

Trust and data boundaries

Publishers receive a JWT or NKey with pub permission on events.> only. Consumers get sub on the same wildcard and consumer.create on their stream. Ops roles hold stream administration permissions. Applications never receive wildcard * permissions. This prevents a compromised publisher from deleting the stream and a consumer from purging messages another group still relies on.

Operational checks

Monitor these signals regularly:

• CONSUMER_LAG per consumer: nats consumer info <stream> <consumer> --count
• STREAM_MSG_COUNT vs max_msgs: nats stream report <stream>
• Disk usage vs max_bytes: server metric jetstream_stream_disk_usage
• REPLICA_LAG: server metric jetstream_replica_lag
• ack_wait expirations and num_pending stagnation: watch for more than 5 expirations per minute or pending not decreasing for more than twice the ack_wait period

Use nats stream report <stream> for a quick health snapshot: it shows message count, byte count, consumer lag, and replica status in one view.

Failure modes and mitigations

Disk full with discard=new

When the byte limit is reached, new publishes return 503 Insufficient Storage. Producers using async publish without awaiting the ack may silently lose messages. Mitigation: always await publish acks (NatsMsg.Ack() in Go, flush() in Java) and implement producer-side backoff with retry.

Consumer crash mid-batch

Unacked messages redeliver after ack_wait. Design idempotent handlers; track message IDs for deduplication if exactly-once is required.

Quorum loss in a 3-node cluster

If two nodes become unreachable, the stream leader steps down; new publishes get 503 No Stream Leader. Reads from the remaining replica continue. Mitigation: deploy across three availability zones; monitor REPLICA_LAG to catch slow replicas before quorum breaks.

Poison message exhausts max_deliver

Without a dead-letter subject, the consumer stalls after max_deliver failures. Always configure a dead-letter subject and alert on messages routed there. Inspect with nats consumer info to view num_redelivered and pending counts.

Conditions that would change the design

Late joiners must never miss recent events

Switch to discard=old with max_age (e.g., --max-age 72h). Producers are never backpressured, but a burst can evict events a slow consumer hasn't fetched.

Geo-redundancy required

Add a mirror stream in a second region: nats stream add <mirror> --subjects 'events.>' --mirror <primary> --storage file --replicas 3. Consumers in the secondary region read from the mirror for low-latency access; failover requires manual promotion.

Dynamic consumer groups (serverless functions)

Migrate the stream to interest retention. Messages disappear once the last consumer acknowledges them, providing auto-cleanup but removing replay capability for new consumers.

High-throughput batch workers

Enable allow_direct on the stream and use pull consumers with large batch sizes: nats stream update <stream> --allow-direct then nats consumer add <stream> <name> --durable --pull --batch 500 --max-wait 5s. Direct get bypasses the consumer's internal queue, reducing latency and CPU overhead for batch-oriented workloads.

Practical verification

After deploying, run a soak test matching peak throughput. Inject failures: kill consumers, pause publishers, isolate a broker. Verify:

• No acknowledged message loss
• Duplicate rates match max_deliver expectations
• Recovery after quorum loss under 30 seconds
• Disk usage stabilizes below max_bytes

If any check fails, adjust ack_wait, max_bytes, or the retention policy before promoting to production.

Limitations and immutable constraints

• Stream configuration (storage, replicas, retention) is immutable after creation. Changing these requires deleting and recreating the stream, which causes data loss unless a mirror or backup exists.
• JetStream requires NATS server 2.10+. Older clusters cannot use these features.
• Clustered streams with replicas=3 need an odd quorum; a 4-node cluster still requires 3 for quorum - the fourth node is a standby.
• Ephemeral push consumers lose all state on disconnect. Use durable pull consumers for any workload requiring offset guarantees.
• discard=new under extreme memory pressure can drop messages producers believe were accepted; verify with explicit publish acks.