Ensuring Message Survival: Using NATS JetStream for Durable Event Logs

In a standard NATS pub/sub model, messages are ephemeral. If a subscriber is offline or the NATS server restarts, any messages sent during that window are lost forever. For telemetry, this is acceptable; for critical business events like order.created, it is a failure. The solution is to implement JetStream, which transforms NATS from a simple message bus into a durable streaming layer.

The Shift from Ephemeral to Durable

Core NATS operates on a "fire and forget" principle. JetStream introduces a persistence layer that writes messages to a durable log on disk before acknowledging receipt to the publisher. This allows the system to provide at-least-once delivery guarantees.

The architecture separates the Stream (the physical storage of messages) from the Consumer (the view or cursor into that storage). While a stream defines how data is stored and replicated, a consumer tracks which messages have been delivered and acknowledged. This separation allows multiple independent services to read the same event stream at their own pace without interfering with one another.

Implementing a Durable Order Stream

To use JetStream, the NATS server must be started with the -js flag or have jetstream: { store_dir: "/path/to/data" } configured in the server configuration file. This requires the server process to have read/write permissions to the specified directory.

First, verify that JetStream is active. Run this command from a terminal with access to the NATS CLI:

nats server info

Check the output for jetstream: enabled. If this is missing, the server is running in core-only mode and cannot persist messages.

Step 1: Define the Stream

Create a stream to capture all order-related subjects. Run this with administrative permissions:

nats stream add ORDER_EVENTS --subjects "orders.>"

In this configuration, ORDER_EVENTS is the stream name, and "orders.>" is a wildcard subject. Any message published to orders.created, orders.shipped, or orders.cancelled will be captured in this single durable log.

Step 2: Publish and Consume Durably

Publish a test event from a producer host:

nats pub orders.created '{"order_id":"12345", "status":"pending"}'

To ensure a worker can resume where it left off after a crash or restart, use a durable consumer. Run this on the worker host:

nats sub orders.created --durable order-processor --stream ORDER_EVENTS

The --durable order-processor flag tells the NATS server to store the state of this consumer's cursor. If the worker disconnects and reconnects using the same durable name, the server will replay all messages that were not yet acknowledged.

Verification and Diagnostics

To verify that messages are actually persisting on disk rather than residing in memory, perform the following check:

1. Publish several messages to the orders.created subject.
2. Stop the NATS server process entirely.
3. Restart the NATS server.
4. Start the durable subscriber again.

If the subscriber immediately receives the messages sent before the restart, persistence is functioning. You can also inspect the current storage usage and message count using:

nats stream info ORDER_EVENTS

Operational Trade-offs

JetStream is not a "free" upgrade; it introduces specific engineering constraints:

• Disk I/O and Space: Unlike core NATS, JetStream consumes disk space. Without a defined retention policy (e.g., --max-age or --max-bytes), a high-volume stream can fill the server's disk, leading to a crash.
• Idempotency Requirements: Because JetStream guarantees at-least-once delivery, network partitions or consumer crashes during acknowledgment can result in the same message being delivered twice. Consumers must be designed to be idempotent (processing the same message twice has no additional effect).
• Complexity: Managing stream limits and consumer lag adds operational overhead compared to the stateless nature of core NATS.