Designing a Minimal Durable Messaging Layer with NATS JetStream

Requirements

Before choosing a design, clarify the functional and non‑functional needs of your messaging layer:

• Durability: Messages must survive a broker restart.
• Delivery guarantee: At‑least‑once is acceptable; exactly‑once must be achieved by the consumer.
• Throughput: Expected publish rate (e.g., 10 k msgs/s) and peak burst.
• Retention: How long messages should be stored (time‑based, size‑based, or interest‑based).
• Fault tolerance: Whether a single broker node loss can be tolerated.
• Security: Isolation of tenants and protection of payloads at rest.

Smallest Suitable Design

For most teams the simplest viable topology is a single NATS server with JetStream enabled and file‑backed storage. No external cluster or separate storage system is required because JetStream is embedded in the nats-server binary.

Configuration snippet

# nats-server.conf
jetstream {
  store_dir = "/var/lib/nats/jetstream"
  max_mem   = 256MB
  max_file  = 5GB
}
authorization {
  # Example: enable accounts for tenant isolation
  accounts: {
    ACME: { users: [ { user: "alice", password: "secret" } ] }
  }
}

Run the server with:

# Run as a user that can write to the store_dir
nats-server -c nats-server.conf

Verify JetStream is active:

# Anywhere with the nats CLI installed
nats server info | grep JetStream

Expected output includes JetStream: true.

Trust and Data Boundaries

NATS provides multi‑tenancy through accounts (or JWT/nsc). Each account isolates connections, subjects, and JetStream streams. However, JetStream persists messages unencrypted on disk by default.

To protect sensitive payloads:

• Enable filesystem‑level encryption (e.g., LUKS, BitLocker) on the store_dir.
• Or encrypt the payload at the application level before publishing and decrypt after receipt.

Check that the store directory is mounted on an encrypted volume:

# Linux example
mount | grep /var/lib/nats/jetstream

Look for a device mapper crypt entry or similar.

Operational Checks

Monitoring JetStream health focuses on stream and consumer state, plus disk usage.

Key metrics to watch

• stream.state.messages – total messages stored.
• stream.state.bytes – disk usage.
• consumer.num_pending – messages delivered but not yet acked.
• consumer.num_redelivered – indicates AckWait timeouts.
• stream.limit.max_bytes vs. actual usage – early warning of disk exhaustion.

Example inspection commands

# Replace MY_STREAM and MY_CONSUMER with your actual names
nats stream info MY_STREAM
nats consumer info MY_STREAM MY_CONSUMER

Look for:

• ACK Floor – last acknowledged sequence; lag here signals slow consumers.
• Redeliveries – if non‑zero, verify consumer idempotence.

Set up alerts when pending count exceeds a threshold (e.g., 10 % of max inflight) or when disk usage reaches 80 % of max_file.

Failure Modes

Understanding what can go wrong helps you size the system and design idempotent consumers.

Disk exhaustion

When a stream hits its max_bytes or max_age limit, further publishes are rejected with a NATSServerError: Stream max bytes exceeded response. Producers must handle this error (e.g., back‑off or dead‑letter).

Consumer ack timeouts

If a consumer fails to ack within AckWait, the message is redelivered after the wait expires. This can cause duplicate processing; consumers must be idempotent or use deduplication via the Nats-Msg-Id header within the stream’s duplicate window.

Clustered mode quorum loss

With a 3‑node JetStream cluster (R=3), losing two nodes removes quorum, pausing leader election and blocking new writes until quorum is restored. Monitor nats server info for JetStream Leader status.

Conditions That Would Change the Design

Revisit the architecture when any of the following occur:

• Throughput exceeds the single‑node storage I/O capacity (sustained > 70 % disk utilization under load).
• Fault‑tolerance requirement evolves from “tolerate a broker restart” to “tolerate a broker node loss without downtime”.
• Regulatory mandates demand encryption at rest and the filesystem cannot be encrypted (e.g., certain cloud‑managed block storage).
• Message semantics shift from at‑least‑once to exactly‑once and you cannot make consumers idempotent; you would need to add an external deduplication store or switch to a different broker.
• Retention policy must change from limits to work‑queue or interest, requiring stream recreation with different max_consumers or discard settings.

When a trigger appears, repeat the steps: re‑evaluate requirements, adjust the smallest suitable design (e.g., move to a 3‑node cluster with R=3, enable server‑side TLS, or switch to a work‑queue policy), and validate with the operational checks above.