Selecting a Jaeger Storage Backend: In-Memory, Elasticsearch, and Cassandra

The Persistence Problem in Distributed Tracing

When deploying Jaeger, the most critical architectural decision is selecting the storage backend. The default configuration uses an in-memory store, which is volatile and local to a single process. In a production environment with multiple service instances and scaled collectors, in-memory storage results in fragmented traces; a request spanning three services may have its spans distributed across three different collectors, making it impossible to reconstruct the full trace in the UI.

The goal is to move from a volatile, single-node setup to a persistent, distributed store that allows the Jaeger Query service to aggregate spans regardless of which collector received them.

Storage Option Comparison

Feature	In-Memory	Elasticsearch	Cassandra
Persistence	None (Volatile)	Persistent	Persistent
Scalability	Single Node	Horizontal (Shards)	Linear (Nodes)
Search Capability	Basic	Advanced Full-Text	Key-based / Limited
Write Throughput	Very High (Local)	High	Extreme
Ops Overhead	Zero	Moderate	High

Trade-offs and Decision Drivers

In-Memory: Local Development Only

Use this for initial integration tests or local development. Because it lacks a shared database, you cannot run multiple Jaeger Collectors behind a load balancer. If you do, your traces will be split across pods, and the UI will show incomplete traces.

Elasticsearch: The General Purpose Choice

Elasticsearch is the industry standard for Jaeger because of its indexing capabilities. When performing root cause analysis, you often need to filter by specific tags (e.g., http.status_code=500 or customer_id=123). Elasticsearch indexes these tags, allowing for near-instant retrieval of specific traces among millions of entries.

Risk: Index bloat. Without a proper Index Lifecycle Management (ILM) policy, Elasticsearch disks will fill rapidly as trace volume grows.

Cassandra: High-Volume Ingestion

Cassandra is optimized for write-heavy workloads. If your system generates an overwhelming volume of spans and you primarily retrieve traces by Trace ID rather than complex tag searches, Cassandra provides superior write performance and linear scaling.

Risk: Operational complexity. Managing a Cassandra cluster requires more specialized knowledge regarding compaction and tombstone management than managing Elasticsearch.

Implementation: Configuring Elasticsearch

To transition from the "all-in-one" development image to a production-ready Elasticsearch backend, you must configure the Jaeger Collector and Query services. This example assumes a Jaeger version 1.x deployment via environment variables.

Configuration Steps

1. Deploy Elasticsearch: Ensure an Elasticsearch cluster is accessible. For production, use a managed service or a cluster with at least three nodes for high availability.
2. Set Environment Variables: Run the following variables on both the jaeger-collector and jaeger-query containers:

# Define the storage type
export STORAGE_TYPE=elasticsearch

# Define the connection string (comma-separated list of hosts)
export ES_SERVERS=http://elasticsearch-cluster:9200

# Define the index prefix to avoid collisions with other data
export ES_INDEX_PREFIX=jaeger

Verification and Diagnostics

After deploying the configuration, verify the connection using these steps:

• UI Connectivity: Open the Jaeger UI. If the backend is misconfigured, the search page will return a 500 error or a "connection refused" message when attempting to load traces.
• Index Validation: Run a GET request against the Elasticsearch API to ensure indices are being created: curl -X GET "http://elasticsearch-cluster:9200/_cat/indices?v". You should see indices matching your ES_INDEX_PREFIX.
• Trace Retrieval: Trigger a request in your application, find the Trace ID in your logs, and search for that specific ID in the Jaeger UI. If the trace appears with all spans intact across multiple services, the distributed storage is functioning correctly.

Rollback Procedure

If the Elasticsearch cluster becomes unstable, you can revert to in-memory storage to restore basic visibility (though you will lose persistence and distributed aggregation). Update the environment variables on the containers:

export STORAGE_TYPE=memory
# Remove ES_SERVERs and ES_INDEX_PREFIX

Restart the pods to apply the change.