Choosing the Right Jaeger Sampling Strategy for Production Traces

Problem: How to Sample Traces in High‑Traffic Microservices

In a busy microservice environment you can generate millions of traces per hour. Storing every trace is impossible, yet dropping too many hides bugs. Jaeger offers three built‑in samplers that balance latency, storage, and fidelity: probabilistic, rate‑limiting, and adaptive. Choosing the right one depends on traffic volume, latency tolerance, and the need for dynamic tuning.

Decision Guideline

Use rate‑limiting when:

• Trace volume is high (hundreds of thousands per second).
• Latency must stay within strict SLAs.
• Deterministic limits per second are required.

Use probabilistic when:

• Trace volume is moderate or low.
• You prefer a simple, stateless approach.
• Fidelity trade‑offs are acceptable.

Use adaptive when:

• Workload is variable and you want the system to self‑tune.
• Jaeger 1.30+ and a collector that supports adaptive sampling are in place.
• Some latency impact is tolerable for higher fidelity during peaks.

Constraints to Consider

• Statelessness: Probabilistic sampling requires no shared state, making it agent‑side only.
• Global Counter: Rate‑limiting and adaptive need a counter shared across agents, so they must run on the collector.
• Latency Impact: Rate‑limiting introduces a small sync cost; adaptive can add jitter during threshold updates.
• Trace Fidelity: Probabilistic drops a fixed fraction; rate‑limiting drops only when the counter exceeds the limit.
• Backend Compatibility: Adaptive sampling is only supported by Jaeger 1.30+ collectors and compatible backends.

Option Comparison Table

Strategy	Stateless	Global Counter	Latency Impact	Trace Fidelity	Typical Use Case
Probabilistic	Yes	No	Low	Low	Low‑volume diagnostics
Rate‑Limiting	No	Yes	Medium	Medium	High‑volume production
Adaptive	No	Yes	Medium‑High	Medium‑High	Variable load, self‑tuning

Concrete Implementation

Probabilistic Sampler

Configure the Jaeger agent (or sidecar) with a fixed probability. For example, to keep 1% of traces:

# jaeger-agent.yml
sampler.type: probabilistic
sampler.param: 0.01

Apply the same config to all agents. No collector change is needed.

Rate‑Limiting Sampler

Set a maximum number of traces per second. The collector maintains a global counter and drops excess traces:

# jaeger-agent.yml
sampler.type: rate_limiter
sampler.param: 1000   # 1,000 traces per second

Ensure the collector is configured to expose the rate_limiter sampler. The agent forwards every trace to the collector, which decides to keep or drop it.

Adaptive Sampler

Requires Jaeger 1.30+ and a collector that supports adaptive sampling. The sampler adjusts the rate based on real‑time traffic and downstream storage capacity:

# jaeger-agent.yml
sampler.type: adaptive
sampler.param: 1000   # Target traces per second

The collector periodically updates the threshold; the agent receives new parameters via the collector_sampling_rate metric.

Verification and Validation

1. Generate Synthetic Load:

   # jaeger-injector --service test-service --count 10000
   

   Run this against a staging environment with the chosen sampler.

2. Check Trace Count:

   # curl http://jaeger-query:16686/api/traces?service=test-service | jq '.data | length'
   

   Compare the actual number of traces to the expected sampling rate.

3. Measure Latency:

   # ab -n 1000 -c 100 http://your-service/endpoint
   

   Run the benchmark before and after enabling the sampler. Verify that average latency stays within SLA.

4. Inspect Collector Metrics:

   # curl http://jaeger-collector:14269/metrics | grep collector_sampling
   

   Look for collector_sampling_rate and collector_sampling_last_update to confirm the sampler’s dynamic behavior.

Trade‑Off Summary

• Probabilistic is the simplest and least intrusive but offers no guarantee on trace count; useful for troubleshooting.
• Rate‑Limiting guarantees a hard cap but requires collector‑side state and can drop traces during spikes, potentially hiding issues.
• Adaptive provides near‑optimal sampling under load shifts but adds complexity and a modest latency penalty during threshold recalibration.

Limitations & Practical Checks

• Rate‑limiting can mask problems that only occur under high load if the counter is saturated.
• Adaptive sampling may introduce jitter if the collector’s update frequency is too low.
• Always test in a staging environment with realistic traffic patterns before rolling out to production.
• Use monitoring dashboards (Prometheus + Grafana) to watch collector_sampling_rate and collector_sampling_last_update in real time.

Conclusion

Selecting the right Jaeger sampler is a balancing act between trace fidelity, storage cost, and latency. Start with probabilistic for low‑volume diagnostics, switch to rate‑limiting when you need a hard cap on trace throughput, and consider adaptive if your workloads fluctuate wildly and you want the system to self‑tune. Validate each choice with synthetic load and latency benchmarks to ensure your observability stack meets production SLAs.