Taming Trace Volume with Probability‑Based Sampling in Jaeger

Why the Volume Problem Exists

In a typical microservice stack, each request can generate dozens of spans. A single 10‑minute burst of traffic can produce millions of spans, each of which is stored in a backend such as Elasticsearch or Cassandra. Storage grows linearly with span count, and querying that data becomes expensive. If you keep every trace, you quickly hit budget limits and degrade query performance.

Enter Probability‑Based Sampling

Jaeger’s collector can decide, on a per‑trace basis, whether to keep a trace. The probability sampler picks a fixed fraction of traces (e.g., 1%) to store. The decision happens *after* the spans have been emitted by the client, so you don’t need to touch application code or modify instrumentation.

Because the sampler is uniform, every trace has the same chance of being stored. Over large volumes this yields a statistically representative sample of system behavior, which is enough for most monitoring and alerting use cases.

Configuring the Collector

Below is a minimal collector.yaml that enables probability sampling. Replace 1.0 with the desired fraction (e.g., 0.05 for 5%).

collector:
  endpoint: "0.0.0.0:14267"
  http_server_options:
    port: 14268
  sampling:
    type: "probabilistic"
    param: 0.01   # 1% probability
  storage:
    type: elasticsearch
    options:
      url: "http://elasticsearch:9200"

Run the collector with this file:

jaeger-collector --config-file collector.yaml

Make sure you have root or a user with write access to the collector’s data directory and that the Elasticsearch endpoint is reachable.

Verifying the Sampler Works

1. Deploy a simple instrumented service (e.g., a Go or Java HTTP server with the Jaeger client).
2. Send a burst of traffic (hundreds of requests) while the collector is running.
3. Open the Jaeger UI and filter by the service name. You should see roughly 1 % of the emitted traces.
4. Check collector logs for lines like Sampled: true/false to confirm the sampler is making decisions.
5. Increase param to 0.05 and repeat the test. The number of stored traces should grow proportionally.

These steps are a quick sanity check. For production, you’ll want automated metrics (e.g., jaeger_collector_sampling_rate) to monitor sampling efficacy.

Trade‑Offs and Limitations

• Missing Rare Events: With a low sample rate, infrequent anomalies may slip through the cracks. If you need to detect flash events, consider a higher rate or an adaptive sampler.
• Uniform Traffic Assumption: The sampler treats all traces equally. During traffic spikes, the absolute number of stored traces may still surge, potentially stressing storage.
• Cost vs Fidelity: Lower rates reduce storage and query costs but also reduce the resolution of your observability data. Find the sweet spot that balances budget and monitoring needs.

Actionable Takeaways

• Start with a 1 % probability sampler in the collector for high‑volume environments.
• Expose the sampling section to a REST API (Jaeger supports /api/sampling) so you can adjust the rate without redeploying the collector.
• Instrument your dashboards to show the sampling rate and the number of stored traces versus emitted traces.
• Periodically review the sampling configuration against traffic patterns and adjust upward if you notice missing critical events.

By configuring probability sampling at the collector, you keep your observability stack lean, avoid code changes, and maintain a clear view of system health. Adjust the rate carefully, monitor the results, and you’ll have a cost‑effective tracing solution that scales with your microservices.