Choosing a Jaeger Storage Backend: In-Memory vs. Elasticsearch vs. Cassandra

The Storage Decision: Volatility vs. Scalability

When deploying Jaeger, the most critical architectural decision is selecting the storage backend. The wrong choice leads to either immediate data loss during pod restarts or an operational burden that outweighs the benefits of distributed tracing. The primary trade-off is between operational simplicity (zero-config) and data durability (persistent indexing).

Backend Comparison Matrix

Feature	In-Memory	Elasticsearch	Cassandra
Durability	Volatile (Lost on restart)	Persistent	Persistent
Query Capability	Basic	Advanced Full-Text/Tags	Key-based/Limited
Write Throughput	Very High (RAM speed)	High (Indexed)	Very High (Linear)
Ops Overhead	None	Moderate to High	High
Best Use Case	Local Dev / CI	Production General Purpose	Extreme Write Volumes

Evaluating the Trade-offs

In-Memory Storage is the default for the Jaeger "all-in-one" binary. It requires no external dependencies, making it ideal for developers testing instrumentation locally. However, because it stores spans in RAM, it cannot handle large volumes of data and wipes all traces whenever the Jaeger collector or query service restarts.

Elasticsearch is the industry standard for Jaeger production environments. It provides powerful indexing, allowing SREs to search for traces based on specific tags (e.g., http.status_code=500) or custom logs. The cost is operational complexity: you must manage JVM heap sizes and disk I/O to prevent the cluster from locking up during heavy write bursts.

Cassandra is designed for high-availability and linear scalability. If your system generates millions of spans per second and you primarily need to retrieve traces by Trace ID rather than performing complex tag-based searches, Cassandra's write-optimized architecture is superior. However, it requires a rigorous compaction strategy to handle the time-series nature of trace data without causing disk fragmentation.

Implementation: Switching to Elasticsearch

To move from a development setup to a persistent Elasticsearch backend, you must modify the Jaeger Collector and Query environment variables. This example assumes a deployed Elasticsearch cluster accessible via HTTP.

Configuration Steps:

1. Ensure the Elasticsearch cluster is running and the jaeger-traces index can be created.
2. Set the following environment variables on the Jaeger Collector and Query pods:

# Run these as environment variables in your Kubernetes Deployment or Docker Compose file
STORAGE_TYPE=elasticsearch
ES_SERVER_URL=http://elasticsearch-cluster:9200
# Optional: Define a specific index prefix to avoid collisions
ES_INDEX_PREFIX=jaeger-production

Verification Process:

• Step 1: Send a test trace using your application's Jaeger client.
• Step 2: Open the Jaeger UI and verify the trace appears in the search results.
• Step 3: Restart the Jaeger Collector/Query pods.
• Step 4: Refresh the UI. If the trace persists, the backend is correctly configured. If it disappears, the system has reverted to in-memory storage or cannot connect to Elasticsearch.

Operational Limitations

Regardless of the backend, Jaeger does not provide built-in data retention policies in the same way a database does. For Elasticsearch, you must implement Index Lifecycle Management (ILM) to automatically delete old indices; otherwise, your disk will fill up, causing the cluster to enter a read-only state.

For Cassandra, you must monitor the gc_grace_seconds and compaction throughput to ensure that the deletion of old spans (TTL) does not cause "tombstone" build-up, which can severely degrade read performance.

Rollback Procedure

If the persistent backend causes latency issues or stability problems, you can revert to in-memory storage by removing the STORAGE_TYPE and ES_SERVER_URL variables and restarting the services. Note that this will result in the immediate loss of all historical trace data stored in RAM.