Using Jaeger Adaptive Sampling to Control Trace Volume in High‑Traffic Services

Problem: trace explosion under load

When a microservice handles thousands of requests per second, each request generates a trace with dozens of spans. In a Jaeger deployment without sampling, the backend can ingest millions of spans each second, quickly filling storage and increasing query latency.

How Jaeger adaptive sampling works

The Jaeger client library exposes a /sampling HTTP endpoint on the agent. The agent periodically fetches a sampling strategy from this endpoint. The strategy contains a base probability and an adaptive controller that raises the probability when the observed request rate spikes, then lowers it again when traffic subsides. This feedback loop keeps the stored span volume roughly proportional to the configured budget while still giving higher visibility to bursty traffic.

Configuration steps

1. Set the Jaeger client environment variables (replace jaeger-agent with your agent host):

   export JAEGER_SAMPLER_TYPE=probabilistic
   export JAEGER_SAMPLER_PARAM=0.001   # base rate
   export JAEGER_SAMPLER_MANAGER_HOST_PORT=jaeger-agent:5778
   

2. Start the Jaeger agent with debug logging to see strategy exchange:

   jaeger-agent \
     --logging.level=debug \
     --collector.host-port=jaeger-collector:14250
   

3. Verify that the agent receives a strategy. In the agent log you should see a line similar to: Received sampling strategy: {probabilistic:0.001, adaptive:{lowerBound:0.0005, upperBound:0.02}}

Worked example

Assume a service receives a steady 10 000 requests / s. With the base rate JAEGER_SAMPLER_PARAM=0.001 the client would sample about 10 000 × 0.001 = 10 traces per second. During a traffic burst the request rate jumps to 50 000 / s for 30 seconds. The adaptive controller raises the probability to the configured upper bound (e.g., 0.01), yielding roughly 50 000 × 0.01 = 500 traces per second during the burst. After the burst the probability decays back toward the base rate. Over a minute the total stored spans increase roughly linearly with the average request rate, preventing a sudden storage spike while still capturing a higher fraction of the bursty traffic for latency analysis.

Trade‑off and mitigation

The main risk is that an overly aggressive lower bound can suppress rare error traces, making them invisible in Jaeger. To mitigate:

• Set a non‑zero lower bound via the agent configuration (e.g., --sampling.lower-bound=0.0005) so that even low‑traffic periods retain a minimal sampling probability.
• Enable Jaeger’s built‑in error‑rate alerts on the error_count metric; if the alert fires while the sampling rate is low, consider lowering the bound or increasing the base rate.
• Periodically review the sampling_rate metric in the Jaeger UI to ensure it stays within your expected band.

Actionable checklist

• Deploy the Jaeger agent with adaptive sampling enabled (version 1.20 or newer).
• Configure the client as shown above, pointing JAEGER_SAMPLER_MANAGER_HOST_PORT at the agent.
• Start the agent with --logging.level=debug and confirm the Received sampling strategy log line appears.
• In the Jaeger UI, open the Metrics tab for your service and watch the sampling_rate gauge; it should rise during load spikes and fall afterward.
• Run a representative load test, record the total span count stored before and after enabling adaptive sampling, and verify that the stored volume aligns with your storage budget.
• If you notice missing error traces, lower the lower-bound or increase JAEGER_SAMPLER_PARAM and repeat the verification steps.