Using NATS JetStream File Storage as a Reliable Event Store in Micro‑services

Requirements

Micro‑services that need low‑latency publish/subscribe with durable replay, at‑least‑once delivery, horizontal scalability, and long‑term audit retention often reach for a dedicated event store. NATS JetStream satisfies these requirements without an external database when the following are true:

• Message latency must stay in the low‑millisecond range.
• Consumers need to replay historic events after a restart or new deployment.
• The system can tolerate at‑least‑once semantics (duplicates are handled idempotently).
• Operational simplicity favors a single runtime (NATS) over a separate log store.

Smallest Suitable Design

A three‑node NATS cluster with JetStream enabled provides high availability while keeping the topology minimal.

1. Cluster: 3 nodes, each running nats-server with jetstream { store_dir: "/var/lib/nats/jetstream" } on local SSD.
2. TLS & Authentication: Mutual TLS for inter‑node gossip; clients authenticate with NKey (or JWT) and are authorized per subject.
3. Stream Definition: One stream per business domain, e.g. ORDERS, configured with max_age: 30d, max_bytes: 500GB, storage: file.
4. Consumers: Durable pull consumers for batch processing, durable push consumers for low‑latency notification.

# Create the stream (run on any cluster node with admin credentials)
nats stream add ORDERS \
  --subjects "orders.>" \
  --max-age 30d \
  --max-bytes 500GB \
  --storage file \
  --replicas 3

Permissions required: the invoking user must hold a sys or jetstream admin NKey. The command writes to the cluster’s shared JetStream metadata; no direct file‑system access is needed.

Trust and Data Boundaries

• Client ↔️ Cluster: TLS encrypts traffic; NKey public keys are stored in the server’s authorization { users } block. Each service receives a unique NKey pair; the private key never leaves the service process.
• Authorization: publish and subscribe permissions are scoped to subject prefixes (e.g., orders.* for the order service).
• Storage Isolation: JetStream files live under /var/lib/nats/jetstream on each node. The directory is owned by the nats user, mode 0700. No external process should mount or read these files.
• Inter‑Node Communication: Cluster routes use mutual TLS with a separate CA; compromise of a single node’s private key does not expose client credentials.

Operational Checks

Metric	Tool	Alert Threshold
Server RPC latency	nats top or Prometheus nats_server_rpc_latency_seconds	> 5 ms p99
JetStream disk usage	nats stream info ORDERS --json → state.bytes	> 80 % of max_bytes
Consumer lag	nats consumer info ORDERS ORDER_PROCESSOR --json → num_pending	> 10 000 messages
Failed auth attempts	Prometheus nats_server_auth_failed_total	> 5/min
Cluster health	nats server check or nats_cluster_nodes_total	Any node missing

Run nats top on a bastion host with the same TLS credentials as the services. The command reads live metrics from the cluster’s monitoring subject ($SYS.REQ.SERVER.PING.*) and requires a user with system permission.

Failure Modes

• Disk saturation: If write throughput exceeds SSD IOPS, JetStream back‑pressures publishers (NATS returns 503 Service Unavailable). Mitigation: provision NVMe, enable max_bytes limits, or move transient streams to memory storage.
• Node loss: With three replicas, the cluster tolerates one node down. Publishers continue to succeed; consumers may see a brief lag while the remaining nodes replicate missing segments. When the node restarts, it streams missing blocks from peers (verified via nats stream info ORDERS showing replicas: 3 again).
• Stream reconfiguration: Changing max_age or max_msgs after creation requires nats stream edit which may briefly pause the stream. Plan versioned stream names (e.g., ORDERS_v2) and migrate consumers in a blue‑green fashion.
• Credential rotation: Rotating NKeys demands updating each service’s key file and the server’s authorization map. Use a short‑lived JWT with a signing key that can be revoked without restarting the server.

When the Design Must Change

• Write throughput consistently exceeds the SSD’s sustained IOPS (observed via iostat -x 1 > 90 % utilization). Consider adding a dedicated JetStream node with memory storage for hot streams.
• Retention requirements grow beyond the cluster’s total disk capacity. Options: tiered storage (offload to S3 via JetStream’s archive feature) or introduce a separate log compaction layer.
• Regulatory audit demands immutable, tamper‑evident logs. JetStream’s file storage is mutable; add a periodic hash‑chain export to an append‑only object store.
• Cross‑region latency becomes a factor. Deploy a second NATS cluster in the target region and use leaf‑node or gateway connections for geo‑replication.

Verification Checklist (Run in Staging)

1. Deploy the 3‑node cluster with TLS + NKey auth.
2. Enable JetStream and create the ORDERS stream as shown above.
3. Publish 10 000 messages: for i in {1..10000}; do nats pub orders.created "{\"id\":$i}"; done
4. Create a durable pull consumer: nats consumer add ORDERS ORDER_PROCESSOR --durable --pull --ack explicit
5. Fetch all messages: nats consumer next ORDER_PROCESSOR --count 10000 and verify payload integrity.
6. Stop one node (systemctl stop nats-server), repeat a publish batch, and confirm the consumer can still retrieve new messages.
7. Restart the stopped node; run nats stream info ORDERS and ensure replicas: 3 and no missing sequence gaps.
8. Monitor disk usage and consumer lag during the test; they must stay within the alert thresholds defined earlier.

Successful completion of the checklist demonstrates that the file‑backed JetStream stream survives node loss, respects configured retention, and provides replayable events for downstream services.