Choosing a Jaeger Storage Backend by Trace Volume and Team Capacity

Decision: pick a storage backend before you deploy Jaeger to production

Jaeger stores collected spans—the individual timed operations that make up a trace—in a backend chosen at collector startup. The wrong choice causes silent data loss, unqueryable indexes, or an operations burden that outgrows the team. This guide helps you choose among the supported Jaeger storage backends based on trace volume, retention needs, and team capacity.

Assumption: this covers Jaeger's native distributed tracing components (jaeger-collector and jaeger-query), not the OpenTelemetry Collector. Configuration keys and defaults vary by Jaeger release; verify them against your installed version before changing production config.

Constraints to define first

• Expected spans per second at peak ingestion.
• Required retention window (days) and whether old traces must be purged automatically.
• Query patterns: trace-by-ID lookups, service-level searches, or tag-based filtering across attributes.
• Team capacity to operate and patch an external cluster.
• Security requirements: encryption at rest, role-based access control, and multi-tenant isolation.

Supported backend comparison

Backend	Best suited for	Scale profile	Retention handling	Operational cost	Key limitation
Memory	Local development, CI, short-lived demos	Single process only	None; data lost on restart	Near zero	Silently drops spans under memory pressure; not production-safe
Badger	Single-node staging or low-to-moderate production volume	One node; embedded disk store	Configurable TTL	Low; no external cluster	Single-writer; multiple collector replicas cannot share the same local store
Cassandra	High-ingest, distributed production environments	Linear write scalability with cluster	TTL plus compaction tuning required	High; cluster capacity planning	Default compaction and TTL settings often need tuning for trace workloads
Elasticsearch / OpenSearch	Workloads needing flexible tag and full-text search	Distributed, shard-dependent	Index lifecycle management required	High; heavy CPU, memory, and disk	Mapping conflicts when span tags use inconsistent types across services
Kafka (buffer only)	Absorbing ingestion spikes in front of any backend	Distributed log layer	Not a long-term trace store	Added pipeline complexity	Increases end-to-end latency; not a storage backend by itself

Trade-offs by scenario

Low volume, single team

Badger keeps spans on disk without running Cassandra or Elasticsearch. It is appropriate when one collector instance can handle the write load and the team does not want to operate a distributed database. Because Badger is embedded and single-writer, you cannot scale the collector horizontally by adding replicas that point to the same Badger directory. A shared network filesystem is not a supported workaround.

High ingestion, many services

Cassandra scales writes linearly and offers tunable consistency, which suits high-span-rate environments. The operational cost is cluster management, capacity planning for the traces and dependencies tables, and compaction tuning. Without TTL and compaction adjustments, storage growth and write latency can degrade over time.

Search-heavy query workloads

Elasticsearch and OpenSearch support flexible tag searches and aggregations, but they consume more resources than Badger or Cassandra. Span attributes with mixed types—for example, the same tag name sent as a string by one service and a number by another—can cause mapping explosions and failed queries. Enforce consistent tag schemas or use strict dynamic mappings before rollout.

Spiky ingestion

Place Kafka between collectors and the storage backend only when ingestion bursts risk overwhelming the backend or causing span loss. Kafka adds operational complexity and latency, so treat it as a buffer, not a replacement for choosing a backend that can sustain your baseline write rate.

Concrete implementation: single-node Badger storage

This example configures jaeger-collector to use Badger on one host. Run the commands on the collector host as the user that runs the Jaeger service, typically a non-root user with write access to the data directory.

Configuration file

# /etc/jaeger/collector-config.yml (example paths; adjust for your installation)
storage:
  type: badger
  options:
    badger:
      directory: /var/lib/jaeger/badger

Where to run and required permissions

• Host: the server running jaeger-collector.
• Permissions: the Jaeger process user must own or write to /var/lib/jaeger/badger. Create the directory and set ownership before starting the collector.
• Command example (run as root or with sudo to set ownership):

sudo install -d -o jaeger -g jaeger -m 0750 /var/lib/jaeger/badger

Expected check: after restarting jaeger-collector, its log should show Badger initialization without permission errors. Query the Jaeger API to confirm the collector is reachable:

curl -s -o /dev/null -w "%{http_code}\n" http://localhost:16686/api/services

Expected check: HTTP 200 and a JSON service list. Do not treat this as proof that Badger is persisting data; verify persistence separately.

Validation that spans survive a restart

1. Send representative trace traffic to the collector while Badger storage is active.
2. Query a known trace ID through jaeger-query or the Jaeger UI.
3. Restart jaeger-collector.
4. Query the same trace ID again.

If the trace is still retrievable after restart, Badger is writing to disk. If it is missing, check collector logs for Badger write errors and confirm the process user owns the data directory.

Risk

Do not start multiple jaeger-collector instances pointing to the same Badger directory. Badger does not support concurrent writers from separate processes, and a shared filesystem is not a supported alternative. For horizontal collector scaling, use Cassandra or Elasticsearch/OpenSearch instead.

Practical verification before rollout

• Run a load test with realistic span cardinality against the chosen backend and measure query latency for trace-by-ID and service-level searches.
• Inspect Jaeger collector logs and backend metrics for dropped spans, write timeouts, or throttling during peak ingestion.
• Validate retention and TTL behavior by confirming old traces are purged and storage size stabilizes after the retention window.
• For Elasticsearch/OpenSearch, check index templates and shard counts against expected daily trace volume before rollout.

Limitations and items for review

This draft is based on established Jaeger storage behavior, but some details require verification against your Jaeger version:

• Exact storage option names and defaults differ between Jaeger releases; check jaeger-collector --help or your release documentation.
• Badger compaction and disk usage characteristics under sustained production load should be validated in your environment.
• Cassandra schema versions and compaction strategies must be matched to your Cassandra release.
• Elasticsearch and OpenSearch compatibility with the Jaeger version in use should be confirmed before production deployment.

Mark these items for review by the team operating the backend, and treat the configuration paths above as placeholders, not tested output.

Rollback note

Changing the storage backend changes system state. Historical traces stored in one backend are not automatically migrated to another. If you switch from Badger to Cassandra, for example, old Badger traces remain in Badger and are not visible to the Cassandra-backed collector unless you perform a separate migration. Plan the cutover as a new deployment with parallel validation, not an in-place upgrade with an instant rollback.