Jaeger Distributed Sampling: Configuring Probabilistic and Remote Samplers in Java

What Distributed Sampling Does

Distributed sampling lets a Jaeger client decide, at trace start, whether to emit the full trace. By dropping a configurable fraction of traces you cut network traffic, storage usage, and CPU load while still keeping a statistically representative view of latency, error rates, and service dependencies.

Configuring the Sampler in Java

The Jaeger Java client reads sampler settings from the opentracing.jaeger.sampler.* properties (or the equivalent YAML/environment variables). The three key keys are:

• sampler.type – probabilistic, rate_limiting, remote, or const.
• sampler.param – numeric value: probability (0.0‑1.0) for probabilistic, traces‑per‑second for rate_limiting, or max‑traces‑per‑second for remote.
• sampler.remote_endpoint – required only for remote; points at the Jaeger collector’s sampling gRPC/HTTP endpoint (default http://jaeger-collector:14250/sampling).

These properties are applied once when the Tracer is built, typically at application start‑up. Changing them requires a restart or a dynamic configuration reload if your framework supports it.

Probabilistic Sampler Example

In a Spring Boot 2.7+ application using the OpenTracing‑Jaeger starter, add the following to application.yml (or application.properties) to keep roughly 1 % of traces:

opentracing:
  jaeger:
    sampler:
      type: probabilistic
      param: 0.01

With this configuration the client generates a random number for each new trace; if the number is below 0.01 the trace is sampled and sent to the collector, otherwise it is dropped locally.

Remote Sampler Example

Centralised sampling lets the collector decide the effective rate for each service. Configure the client to fetch the sampling strategy from the collector:

opentracing:
  jaeger:
    sampler:
      type: remote
      param: 1000
      remote_endpoint: http://jaeger-collector:14250/sampling

param here is the maximum traces‑per‑second the client will request; the collector returns a per‑service probability that the client applies. If the endpoint is unreachable the client silently falls back to a const sampler with param=0 (i.e., no traces), which can change behaviour without an obvious error.

Limits and Common Pitfalls

• Client‑side decision only. Once a trace is dropped it cannot be recovered. Rare but critical events (e.g., a single failed payment) may be missed if the sampling probability is too low.
• Context propagation required. Downstream services only see the sampling decision if the trace context (trace‑id, span‑id, sampling flag) is propagated via HTTP headers or messaging metadata. Services that do not propagate context will start new traces with their own sampler, breaking the end‑to‑end view.
• Remote sampler fallback. An unreachable remote_endpoint causes a silent fallback to a const sampler (often param=0). Monitor collector logs and client metrics to detect this.
• Over‑sampling. Setting param=1.0 for probabilistic or a very high rate for rate_limiting defeats the purpose and can overwhelm the collector and storage.
• Parameter extremes. A param of 0.999 for probabilistic means 99.9 % of traces are dropped – effectively no observability. Verify the intended magnitude.

Verifying Sampling Behaviour

1. Jaeger UI trace rate. In the Jaeger UI, compare the traces‑per‑minute count with the expected rate. For a 1 % probabilistic sampler on a service receiving 10 000 requests/min, you should see ~100 traces/min (within statistical variance).
2. Client metrics. Enable the jaeger-metrics library (Micrometer/Prometheus). The counters jaeger_sampler_sampled and jaeger_sampler_dropped should reflect the configured ratio. A sudden shift indicates a fallback or mis‑configuration.
3. Controlled load test. Send a known number of requests (e.g., 10 000 via hey or JMeter) and count recorded traces in the UI or storage backend. The observed count should match the expected sampling ratio (≈100 for 1 %).
4. Collector logs. Check the Jaeger collector logs for errors like "failed to fetch sampling strategy" or connection timeouts to the remote sampling endpoint. Absence of such errors confirms the remote sampler is reachable.

Version Assumptions

Examples assume Jaeger Java client 1.8+, OpenTracing‑Jaeger starter 2.3+, and Spring Boot 2.7+. The property names are stable across these versions; newer OpenTelemetry‑based instrumentation uses a different configuration model.