Picking a Jaeger Sampling Strategy Without Drowning Your Storage

Your tracing bill is a sampling decision. Every span your services emit has to travel over the network, get processed by the collector, and sit in storage until retention kicks in. At low traffic that's fine. At real production traffic, storing 100% of traces is usually the first thing that breaks the budget — and the fix, choosing a sampling strategy, is less obvious than it looks because Jaeger gives you four options with genuinely different behavior.

The thesis of this post: start with head-based probabilistic sampling, and graduate to adaptive (remote) sampling only when you have a per-endpoint imbalance problem. Tail-based sampling is a different architecture entirely, and Jaeger won't do it for you.

Head-based sampling: decide early, propagate the decision

All of Jaeger's built-in samplers are head-based: the decision to record a trace is made at the very first span, when the trace starts. That decision is then carried in the trace context headers (the uber-trace-id header in classic Jaeger clients, or the traceparent sampled flag in W3C context) so every downstream service honors it. This is what keeps traces complete — you never get a trace that's missing its middle because some intermediate service rolled the dice differently.

The four sampler types:

• Constant — sample everything (param=1) or nothing (param=0). Useful for dev and for forcing traces during debugging.
• Probabilistic — sample a fixed fraction of traces, e.g. 10%. Stateless, simple, no coordination needed.
• Rate-limiting — sample at most N traces per second per service instance. Caps cost, but the fraction varies with traffic.
• Remote — the client asks a Jaeger agent or collector what strategy to use, and polls for updates. This is the foundation of adaptive sampling.

The problem probabilistic sampling can't see

Probabilistic sampling is operation-agnostic. If your service has a health-check endpoint handling 5,000 requests per second and a password-reset endpoint handling two per hour, a 10% sampler treats them identically. You'll store thousands of boring health-check traces and quite possibly never capture a single trace of the rare endpoint — which is exactly the endpoint whose failures you care about.

You can partially patch this in application code (force-sampling on error, or bumping the sample decision when latency exceeds a threshold), but that logic lives in every service and drifts out of sync.

Adaptive sampling: per-operation probabilities from the collector

Adaptive sampling solves the imbalance by moving the decision policy out of the service. With JAEGER_SAMPLER_TYPE=remote, the client fetches a sampling strategy from the agent or collector's sampling endpoint and refreshes it periodically. The collector (with adaptive sampling enabled) observes actual traffic per service and per operation, then computes probabilities that throttle noisy endpoints while keeping rare ones at or near 100% — all without redeploying anything.

A minimal local setup to see the difference, using the all-in-one image (suitable for development, not production storage):

# Terminal 1 — run Jaeger all-in-one (in-memory storage)
docker run --rm -p 16686:16686 -p 6831:6831/udp -p 14250:14250 \
  jaegertracing/all-in-one:latest

# Terminal 2 — run your instrumented demo app with 10% probabilistic sampling
JAEGER_SAMPLER_TYPE=probabilistic \
JAEGER_SAMPLER_PARAM=0.1 \
JAEGER_AGENT_HOST=localhost \
./my-instrumented-service

Run these as a normal user with Docker access; no elevated privileges needed. Generate a few hundred requests against your service, then open the UI at http://localhost:16686 and search for the service. You should see roughly one trace in ten — don't expect an exact ratio on small sample sizes. To check what a remotely-configured client actually received, query the sampling endpoint directly:

curl "http://localhost:14250/api/sampling?service=my-instrumented-service"

(Port and path depend on whether you query the agent or collector and on your Jaeger version — check the release notes for your deployment.) If the response shows a default probabilistic strategy instead of your configured one, the client likely can't reach the endpoint and has silently fallen back to defaults. That silent fallback is the biggest operational gotcha of remote sampling: nothing crashes, you just get different sampling than you think.

What Jaeger won't do: tail-based sampling

Head-based sampling decides before the trace runs, so it can't know the trace contained an error or took nine seconds. If you need "keep all errors and slow traces, sample the rest," that's tail-based sampling, and it requires buffering spans until the trace completes. Jaeger's collectors don't do this. The standard pattern is to put the OpenTelemetry Collector in front: services export spans to it, its tail sampling processor holds spans in memory, applies policies (error status, latency thresholds, attribute matches), and only then forwards survivors to Jaeger.

The trade-off is real: the OTel Collector must see every span of a trace on the same instance to make a correct decision, which means routing by trace ID and paying the memory cost of buffering. It's more moving parts and more failure modes. If per-endpoint throttling solves your problem, adaptive sampling is far cheaper to operate.

A note on client libraries and versions

Sampler configuration keys and behavior are version-sensitive, and the classic Jaeger client libraries have been deprecated in favor of OpenTelemetry SDKs, which configure samplers differently (e.g. OTEL_TRACES_SAMPLER=parentbased_traceidratio rather than JAEGER_SAMPLER_TYPE). Before copying any environment variables from a blog post — including this one — check the Jaeger and OpenTelemetry release notes for the versions you actually run, and verify behavior empirically with the all-in-one setup above.

A practical decision rule

Start with probabilistic head-based sampling at a rate your storage can afford, plus application-level force-sampling for errors. Move to adaptive sampling when you can point to a specific hot-endpoint/rare-endpoint imbalance and you're prepared to monitor the sampling endpoint's reachability. Reach for tail sampling via the OTel Collector only when "we must never lose an error trace" is a hard requirement — because you're signing up to run a stateful buffering pipeline to get it.