Using Jaeger Adaptive Sampling to Keep Trace Volume Under Control

Problem: Trace volume explodes while errors go unseen

When a microservice mesh generates tens of thousands of requests per second, storing every trace quickly overwhelms storage and increases cost. At the same time, you still need to capture enough traces from low‑traffic or failing paths to debug incidents. Static probabilistic sampling either throws away too many error traces or wastes capacity on healthy high‑volume traffic.

Thesis: Adaptive sampling lets the Jaeger collector continuously tune per‑operation sampling probabilities so the overall trace ingest stays near a target TPS while automatically boosting rates for rare or error‑prone operations.

How adaptive sampling works in Jaeger

The Jaeger collector runs a control loop (default 10 s interval) that aggregates per‑operation metrics from incoming spans: span count, error count, and latency buckets. Using these metrics it computes a new sampling probability for each operation that will move the observed trace rate toward a global targetTracesPerSecond. Operations that have recent errors or latency spikes get a higher probability; high‑throughput healthy operations get a lower probability. The resulting strategy is served to agents via gRPC/HTTP, cached locally, and applied to new spans as they start.

If the collector becomes unreachable, agents fall back to a configured default probability (often 0.001) until the collector recovers, which can cause a sudden drop or spike in ingested volume.

Configuring the collector and agent

1. Create a strategies file (JSON or YAML) that defines the adaptive sampling parameters. Example JSON:

   {
     "sampling": {
       "type": "adaptive",
       "options": {
         "targetTracesPerSecond": 100,
         "maxOperations": 500,
         "rateLimitUpdates": 2
       }
     }
   }

   Save this as /etc/jaeger/sampling-strategies.json on the collector host.

2. Start the Jaeger collector with the file and ensure it can bind to the gRPC port (default 14250) and HTTP port (default 14268). Run as a user with read access to the file and permission to open those ports:

   jaeger-collector \
     --sampling.strategies-file=/etc/jaeger/sampling-strategies.json \
     --collector.zipkin.host-port=:9411 \
     --log-level=info
   

3. Configure the Jaeger agent to point at the collector (or use the sidecar pattern). The agent will automatically fetch the strategy via gRPC on startup and refresh it periodically. No extra flags are needed; the agent caches the strategy locally.

Worked example: balancing a health check and a payment endpoint

Assume a service mesh with ~50 services. You set targetTracesPerSecond: 100 in the collector.

• The /health endpoint receives 50 k RPS with virtually no errors. To keep the overall trace rate near 100 TPS, the adaptive algorithm assigns it a probability of roughly 0.002 (≈100 TPS / 50 k RPS).

• The /payment endpoint gets only 5 RPS but 20 % of those requests error. Because it is low‑volume and error‑prone, the algorithm boosts its probability to about 0.5, yielding ~2‑3 error traces per second—enough to spot problems while still respecting the global target.

In practice you would see these probabilities in the collector logs (sampling strategy updated messages) and can verify them by querying the agent:

curl http://localhost:5778/sampling

The returned JSON includes a perOperationProbabilities map where you can confirm the values above.

Trade‑offs and limitations

• Collector dependency: If the collector cluster is unavailable, agents use the fallback probability, which can cause a sudden volume change. Mitigate by running multiple collector replicas with a shared storage backend (Cassandra, Elasticsearch) so the strategy is persisted and available.

• Control‑loop latency: The probability update occurs every ~10 s (configurable). A flash‑sale traffic spike may temporarily exceed the target TPS before the loop reacts. Monitor the traces_per_second metric in Jaeger or your observability system to detect overshoots.

• Very low‑throughput operations: An operation with <1 RPS may receive probability = 1.0, potentially blowing the target if many such ops exist. Use maxOperations to cap the number of distinct operations the algorithm tunes, and consider per‑service overrides in the strategies file to impose a ceiling.

• Sampling decision visibility: The sampling probability is not stored on the span itself. Enable the SPAN_SAMPLING_DECISION tag (Jaeger 1.40+) to record the applied probability on each span, then inspect a few traces in the UI to confirm that error/low‑volume paths have higher rates.

Actionable steps to verify and tune

1. After deploying the collector with the strategies file, check the logs for lines like:

   INFO  sampler.adaptive - sampling strategy updated: targetTPS=100, updatedOps=23

2. Query the agent’s /sampling endpoint and note the ttl field; it shows how long the cached strategy is considered fresh.

3. Generate a known load (e.g., using hey or wrk) against a service with a measured RPS. Compare the observed traces per second in Jaeger’s UI or via the traces_per_second metric to your targetTracesPerSecond. Expect a steady‑state variance of roughly ±20 % due to the control‑loop granularity.

4. If you see consistent overshoot, lower targetTracesPerSecond or increase the control‑loop interval (--sampling.update-interval) to make the system more responsive.

5. To revert to the previous static sampling strategy, simply replace the strategies file with a static definition (e.g., { "type": "probabilistic", "param": 0.001 }) and restart the collector. Agents will pick up the new strategy on their next refresh.