Enabling Jaeger Adaptive Sampling: A Practical Guide for Self‑Adjusting Trace Volume

Desired Outcome

In a large microservice landscape, a single static sampling ratio quickly becomes either too noisy or too sparse. Jaeger’s adaptive sampling lets the collector calculate a probability for each service‑operation pair so that the total number of traces ingested stays close to a target rate, regardless of traffic spikes or quiet periods.

Prerequisites

• A Jaeger deployment with at least one running collector that can reach the configured storage backend.
• All instrumented applications must query the collector (or agent) for a sampling strategy – i.e. use remote or probabilistic with a remote sampler URL.
• For multi‑collector setups, a shared adaptive sampling store (e.g. a PostgreSQL or Cassandra instance) so that all collectors agree on the same probabilities.
• Collector version 1.x that supports the adaptive strategy store; verify with jaeger-collector --help | grep adaptive.

Configuration Steps

1. Set up the shared store (if deploying more than one collector). Example PostgreSQL URL:

   export ADAPTIVE_STORE_URL=postgresql://user:pass@db:5432/jaeger
   

2. Launch the collector with adaptive strategy flags.

   jaeger-collector \
         --sampling-strategy-store=adaptive \
         --sampling-strategy-store-adaptive-target-traces-per-second=100 \
         --sampling-strategy-store-adaptive-delta-tolerance=0.1 \
         --sampling-strategy-store-adaptive-interval=30s \
         --sampling-strategy-store-adaptive-store=postgresql \
         --sampling-strategy-store-adaptive-store-url=$ADAPTIVE_STORE_URL
   

   • target-traces-per-second – the desired ingest rate.
   • delta-tolerance – how far the current rate can deviate before adjusting probabilities.
   • interval – how often the collector recalculates probabilities.
3. Configure clients for remote sampling.

   sampler:
     type: remote
     remoteEndpoint: http://collector:14250/sampling
     timeout: 200ms
     maxTracesPerSecond: 0
   

   Clients will query the collector at startup and after each span to obtain an up‑to‑date probability.

4. Deploy the updated configuration to all collectors and restart them.

Expected Checks

After the collector starts, you should see log lines similar to:

INFO  [adaptive] Calculated sampling probability for service 'orders' operation 'CreateOrder': 0.23

Verify the following:

• Collector metrics – jaeger_sampling_strategy_store_adaptive_count should increment as probabilities are emitted.
• Jaeger UI – the number of traces per second for a busy service should hover around the configured target.
• Client logs – confirm that the sampler is receiving a probability field from the collector.

Concrete Verification Example

1. Send a burst of requests to the orders service (e.g. using ab or a simple load script).
2. In the Jaeger UI, filter by service and check the trace count over a 30‑second window. It should be close to the target (±10%).
3. Inspect the collector’s Prometheus endpoint (/metrics) for the label service="orders" operation="CreateOrder" probability="0.23". This confirms the probability is being applied.

Recovery Options

• Revert to static strategy – change the collector flag --sampling-strategy-store=static and provide a sampling.json file with fixed probabilities.
• Disable adaptive store – remove the adaptive-store-url environment variable and restart collectors; they will fall back to the last known probabilities.
• Adjust target rate – increase or decrease target-traces-per-second if the collector is under‑ or over‑sampling.
• Client fallback – if the collector becomes unreachable, clients default to a probability=0.001 (or whatever is configured in the SDK).

Limitations & Practical Tips

• Adaptive sampling is a client‑side drop – once a span is discarded, it cannot be recovered for debugging.
• Bursty workloads may cause probability oscillations; consider increasing delta-tolerance or interval for smoother behavior.
• Low‑traffic operations can be sampled at 0.0, making them invisible. If you need guaranteed visibility, use a minProbability flag (available in newer Jaeger releases).
• Version differences: flag names and supported parameters change between Jaeger 1.15, 1.20, etc. Always consult jaeger-collector --help for your exact build.
• Monitoring the adaptive metrics (jaeger_sampling_strategy_store_adaptive_*) gives early insight into whether the collector is hitting the target.

Putting It All Together

With the collector running and clients configured for remote sampling, the system will automatically adjust trace probabilities so that your ingestion pipeline stays within capacity while still capturing representative traces for each service. Periodically review the adaptive metrics and UI to ensure the target remains appropriate as traffic patterns evolve.