Jaeger Adaptive Sampling: Stop Drowning in Healthy Traces

The Fixed Sampling Trap

You're running a microservices platform handling 50,000 requests per second. Your Jaeger setup uses a fixed 1% sampling rate. That's 500 traces per second flowing into storage—mostly successful health checks and routine API calls. When a rare payment timeout occurs at 0.001% frequency, you get maybe one trace every few minutes. Finding it feels like searching for a needle in a haystack where the hay keeps multiplying.

Fixed sampling treats all operations equally. High-volume, low-value traffic consumes your storage budget while the traces you actually need—errors, latency outliers, new deployment validations—get discarded by probability.

How Adaptive Sampling Changes the Equation

Jaeger's Adaptive Sampling (available since Jaeger 1.30, stabilized in 1.40+) moves the sampling decision from a static percentage to a dynamic feedback loop. The Jaeger Collector observes incoming span traffic per operation name, calculates a target trace rate, and propagates that rate back to instrumented clients via the sampling manager protocol.

Instead of "sample 1% of everything," you configure: "aim for 10 traces per second for each operation." The collector does the math. If GET /health runs at 20,000 RPS, it gets a 0.05% rate. If POST /payment runs at 5 RPS, it gets 100% (capped at your configured maximum). New, unseen operations start at a high initial rate so you never miss a new code path.

Worked Example: Collector Configuration

Enable Adaptive Sampling in the Jaeger Collector YAML (tested against Jaeger 1.50+):

collector:
  sampling:
    adaptive:
      enabled: true
      target_traces_per_second: 10
      max_traces_per_second: 1000
      min_sampling_rate: 0.001
      initial_sampling_rate: 1.0
      update_interval: 10s

Key parameters explained:

• target_traces_per_second: Desired throughput per operation name. Start with 10–50 for most services.
• max_traces_per_second: Hard ceiling per operation. Prevents a misbehaving endpoint from overwhelming storage.
• min_sampling_rate: Floor (0.1% here). Ensures even massive operations yield some traces.
• initial_sampling_rate: Rate applied to brand-new operation names (1.0 = 100%). Critical for new deployments.
• update_interval: How often the collector recalculates and pushes rates. 10s balances responsiveness with client churn.

Clients (Java, Go, Node, Python) need no code changes if they already use the Jaeger remote sampling protocol (JAEGER_SAMPLER_MANAGER_HOST_PORT pointing at the collector). The collector serves /sampling endpoints that clients poll.

Trade-offs You'll Live With

Propagation delay. The collector observes, calculates, publishes, and clients poll. Expect 10–30 seconds before a rate change reaches your services. During a sudden traffic spike, you'll briefly overshoot your target.

Trace continuity gaps. If service-a samples a request but service-b (different operation name) applies a lower rate, the trace breaks. Mitigate by setting min_sampling_rate high enough for critical paths, or use tail-based sampling downstream for error-focused retention.

Collector dependency. Adaptive Sampling requires the Collector component. Pure Agent → Storage deployments (common in early Jaeger adopters) cannot use it. Plan a Collector rollout if you haven't already.

Cardinality explosion. Each unique operation name gets its own rate bucket. Dynamic route parameters (/user/{id}) create thousands of operations if not normalized. Ensure your instrumentation uses route patterns, not concrete URLs, as operation names.

Verify It Works

1. Deploy the collector with the config above.
2. Generate load: hey -n 100000 -c 50 http://your-service/health
3. Watch the Jaeger UI → Sampling page. The /health rate should drop toward 0.05% within 2–3 update intervals.
4. Deploy a new endpoint. Confirm it appears at 100% (or your initial_sampling_rate) for the first few minutes.
5. Check storage ingestion rate. You should see total traces/second stabilize near target_traces_per_second × active_operations.

If traces for a critical operation vanish, raise its min_sampling_rate or add a static sampling rule for that operation in the collector's strategies file as an override.

Start Small, Tune Incrementally

Adaptive Sampling isn't a silver bullet—it's a feedback controller. Begin with target_traces_per_second: 20 and update_interval: 30s in staging. Observe the rate oscillations for a week. Tighten the interval and lower the target once you trust the loop. Your storage bill (and your on-call engineer) will thank you.