Fine‑Tuning Jaeger with Adaptive Sampling: A Practical Guide

Why Adaptive Sampling Matters in a Microservice Stack

In a typical microservice environment you can generate millions of traces a day. Storing every single trace quickly exhausts disk space, inflates network traffic, and overwhelms the collector. Jaeger’s adaptive sampling feature tackles this problem by discarding low‑value traces while keeping the ones that matter for diagnostics.

How Jaeger Makes Sampling Decisions

Sampling happens inside the Jaeger agent, which sits next to each service process. The agent inspects every span before it leaves the host, so only the chosen traces are sent to the collector. This keeps the collector load predictable and reduces the amount of data that travels across the cluster.

Adaptive sampling is declarative. You can set it in a YAML file or via environment variables. The algorithm can be a simple fixed probability, a rate‑limit, or an auto‑tune mode that reacts to traffic spikes.

Example YAML Configuration

jaeger:
  agent:
    sampling:
      # Fixed probability of 1% (1/100)
      probability: 0.01
      # Or a rate limit of 200 traces per second
      # rateLimit: 200
      # Enable auto‑tuning (requires Jaeger 1.27+)
      # autoTune: true

Place this in jaeger-agent.yaml or export the corresponding environment variables before starting the agent.

Verifying That Sampling Works

After the agent boots, expose its metrics endpoint (default http://agent-host:5775/metrics). Two metrics are most useful:

Metric	Description
jaeger_agent_sampling_rate	Current sampling probability (e.g., 0.01)
jaeger_agent_sampling_rate_total	Total number of traces processed by the agent

Run a quick load test: send 10,000 requests to http://service/api and then query the metrics. If you set a 1% probability, you should see roughly 100 traces recorded by the collector. Compare the trace count in the Jaeger UI with the request count to confirm the ratio.

Practical Trade‑Offs

• Missing Critical Errors: A too‑low sampling rate may drop rare error traces. Always keep a higher probability for error paths or add an error filter that overrides the default.
• Inconsistent Settings: If different services use different samplers, a distributed trace can be partially missing. Keep a common sampler policy across the cluster or use a central configuration management system.
• Latency Spikes: During auto‑tuning, the agent may temporarily increase latency. Monitor jaeger_agent_latency_ms and set an alert if it exceeds a threshold.
• Compliance Constraints: Some industries require full trace retention. In those cases, disable adaptive sampling or restrict it to non‑critical services.

Actionable Checklist for Your Deployment

1. Define a sampling policy that balances cost and observability. Start with a 0.5% probability for all services.
2. Deploy the Jaeger agent with the sampling section in its YAML file.
3. Run a short traffic burst and verify jaeger_agent_sampling_rate matches the configured value.
4. Check the Jaeger UI for trace counts versus request counts. Aim for a consistent ratio.
5. Set up Prometheus alerts on jaeger_agent_sampling_rate and jaeger_agent_latency_ms.
6. Periodically review error traces. If you notice missing errors, increase the probability or add a dedicated error sampler.
7. Document the sampler configuration in your observability playbook so new services inherit the same policy.

By following these steps you can keep Jaeger lightweight while still capturing the traces that matter most. Adaptive sampling is a powerful tool, but like any optimization it requires careful tuning and continuous monitoring.