Choosing NATS JetStream File Storage for Durable Cloud-Native Messaging

Problem: needing durability without adding a separate database

Many cloud-native services rely on NATS for low-latency pub/sub, but a pure in-memory JetStream stream loses messages when the server restarts. Adding an external persistence layer (for example PostgreSQL or a distributed log) increases operational complexity and introduces another failure domain. The question is whether NATS JetStream's built-in file-based store can provide the durability we need while keeping the deployment simple.

Thesis: file store is a practical, low-overhead durability option

JetStream file store offers a practical way to persist messages on modest SSD hardware, provided you monitor disk I/O and configure retention wisely. It is not a magic durability layer; it trades a small amount of latency for restart survival, and it demands the same operational attention you would give any disk-backed system.

How the file store works

When a JetStream stream is defined with store: File, NATS writes each message to an append-only log on disk. The log is segmented, and the server replays it on startup to rebuild the stream state. Consumers track their position via file offsets, so durable subscriptions survive restarts. No external database is required; the same NATS binary handles clustering, JetStream, and storage.

Configuration basics

# Add a stream that persists to file (run on any host with nats CLI access)
nats stream add orders \
  --subject 'orders.>' \
  --storage file \
  --max_bytes 5GB \
  --max_age 72h

The command above creates a stream named orders that:

• Stores messages in the file store (--storage file)
• Limits total disk usage to 5 GB (--max_bytes)
• Retains each message for at most 72 hours (--max_age)

You can adjust max_bytes and max_age later with nats stream update. The defaults (no limits) allow unlimited growth, which is dangerous on constrained disks. The NATS server process needs read/write access to its JetStream storage directory.

Worked example: verifying persistence across a restart

1. Create the stream as shown above.
2. Publish a burst of test messages from any machine with the nats CLI and network access to the server:

for i in $(seq 1 10000); do
  nats pub orders.test "payload-$i"
done

1. Consume with a durable subscriber in a separate terminal, acknowledging each message:

nats sub orders.test --durable order_worker --ack

1. Stop the NATS server (for example systemctl stop nats or by killing the process), then start it again. The server logs should contain a line indicating that JetStream is restoring the stream from the file store; the exact wording varies by version, so treat any log excerpt as illustrative rather than guaranteed output.
2. Verify durability: the durable consumer should resume from its last acknowledged offset. Check with nats consumer info orders order_worker and inspect the pending and delivered counts. You can also run nats server report to confirm the stream's stored byte count survived the restart.

This procedure changes state on the server (it creates a stream and writes files), so clean up afterwards with nats stream rm orders if it was only a test.

Trade-offs and limitations

• Latency vs. throughput: file store adds roughly 1-2 ms of write latency compared to a pure memory store, but on a typical SSD you can still achieve hundreds of thousands of messages per second. If your application requires sub-millisecond publish latency, consider keeping a small hot path in memory and archiving older data to a file-backed stream.
• Disk I/O pressure: sustained high write rates can saturate the underlying storage, causing back-pressure that raises publish latency or stalls connections. Monitor disk latency (for example via iostat or cloud provider metrics) and ensure the SSD's provisioned IOPS exceed your peak write rate.
• No automatic compaction: the append-only log retains all messages until the stream's retention policy (size or age) triggers deletion. Mis-configured policies can cause unexpected disk growth. Regularly review nats server report output for JetStream store metrics and set alerts on stored bytes approaching your max_bytes limit.

Actionable closing

If you need durable messaging without adding extra infrastructure, NATS JetStream's file store is a solid starting point:

1. Define streams with --storage file and set explicit max_bytes and max_age limits.
2. Deploy on SSDs with sufficient IOPS; monitor disk latency and JetStream store metrics.
3. Use durable consumers to track processing progress, and verify survival by actually restarting the server in a test environment before relying on it in production.
4. Review retention settings periodically to prevent runaway disk usage.

By treating the file store as a configured component rather than a black box, you gain JetStream's durability benefits while keeping operational overhead low.