Tuning Jaeger Sampling in Production: Balancing Trace Fidelity and Performance

Desired Outcome

In a high‑throughput microservices stack, you want enough trace data to spot latency spikes and bugs, but you cannot afford to flood the Jaeger backend with every request. The goal of this guide is to show how to configure Jaeger’s sampling strategy so that trace volume stays within storage and network limits while still delivering a representative view of application behavior.

Prerequisites

• Jaeger deployed in the cluster (agent, collector, UI, and optional Prometheus metrics endpoint).
• Access to the Jaeger configuration (YAML for the operator or environment variables for Helm/stand‑alone deployments).
• Prometheus or another metrics sink to scrape jaeger_sampler_* metrics.
• Basic understanding of your service traffic patterns (average QPS, expected burst size).

Step 1: Choose a Sampling Strategy

Jaeger offers four built‑in samplers:

• Const – always sample or never sample, useful for debugging.
• Probabilistic – samples each request with a fixed probability p, ideal for steady‑state traffic.
• Rate Limiting – caps the number of traces per second, protects against traffic spikes.
• User‑Defined – custom logic via a plugin (rare in production).

For most production environments a combination of Probabilistic (global default) and Rate Limiting (per‑service override) gives the best trade‑off.

Probabilistic Example

sampler.type: probabilistic
sampler.param: 0.01  # 1% of requests will be traced

Rate Limiting Example

sampler.type: ratelimiting
sampler.param: 500  # max 500 traces per second

Step 2: Configure the Global Default

Set the global sampler in the Jaeger agent or collector configuration. If you use the Jaeger Operator, add the following to the Jaeger custom resource:

spec:
  strategy: allInOne
  allInOne:
    image: jaegertracing/all-in-one:1.47
  config:
    sampler:
      type: probabilistic
      param: 0.01

For a Helm release, set the values in values.yaml:

agent:
  config:
    sampler:
      type: probabilistic
      param: 0.01

Deploy the change and wait for the pod to restart.

Step 3: Override per Service

When a service experiences high traffic bursts, you can cap its trace rate without affecting the rest of the system. Add an annotation to the deployment or pod spec:

metadata:
  annotations:
    jaegertracing.io/sampler.type: ratelimiting
    jaegertracing.io/sampler.param: "200"

This tells the Jaeger client library to use a rate‑limiting sampler that allows at most 200 traces per second from that service. The annotation is read by the client at startup, so no code changes are required.

Step 4: Verify Sampling Behavior

After redeploying, validate that the sampler behaves as expected.

1. Check UI trace count. Generate a burst of traffic (e.g., using hey or wrk) and count the number of traces that appear in the Jaeger UI over a 5‑minute window. It should be close to the configured probability or limit.
2. Scrape sampler metrics. Jaeger exposes jaeger_sampler_* metrics. Example Prometheus query:

   jaeger_sampler_decision_total{sampler_type="probabilistic"}
   

   This metric increments each time a sampling decision is made. Divide the number of accepted traces by the total decisions to approximate the effective probability.
3. Rate‑limit check. For a service with a rate‑limiting override, query:

   jaeger_sampler_rate_limit{sampler_type="ratelimiting"}
   

   The value should not exceed the configured sampler.param even during traffic spikes.

Example verification script (bash + curl):

#!/usr/bin/env bash
set -e
export PROMETHEUS_URL=http://prometheus:9090/api/v1/query

# Probability check
prob=$(curl -s "$PROMETHEUS_URL?query=jaeger_sampler_decision_total{sampler_type=\"probabilistic\"}" | jq -r '.data.result[0].value[1]')
accepted=$(curl -s "$PROMETHEUS_URL?query=jaeger_sampler_decision_total{sampler_type=\"probabilistic\",decision=\"sampled\"}" | jq -r '.data.result[0].value[1]')
ratio=$(awk "BEGIN{print $accepted/$prob}")
echo "Effective sampling ratio: $ratio"

Adjust the sampler.param if the ratio is outside the target ±10% range.

Step 5: Rollback Plan

If the new configuration causes trace volume to spike unexpectedly, you can revert quickly:

• Restore the previous sampler.param in the Jaeger configuration or annotations.
• Restart the affected pods (the agent or the service pods).
• Monitor the jaeger_sampler_* metrics to confirm the volume has returned to acceptable levels.

Because sampling decisions are made at the client or agent level, changing the sampler does not alter existing traces. In‑flight traces may still be emitted, but the overall system stability is preserved.

Limitations & Practical Checks

• Probabilistic sampling is statistical; short bursts may deviate from the target ratio. Use a longer observation window for accurate assessment.
• Rate‑limiting may discard traces that would otherwise be useful during spikes. Consider combining it with a higher Probabilistic rate for critical services.
• Annotations are read at pod start. Rolling updates that change annotations may leave a short window where the old sampler is still active.
• Jaeger’s default Const sampler is useful for debugging but should not be used in production due to data volume.

To validate the final setup, run a realistic load test against the entire service mesh and compare the trace count with the expected theoretical value (traffic × sampling probability or limit). If the observed count is consistently below the target, increase the probability or limit; if it is above, reduce them.