Probability‑Based Sampling in Jaeger: How to Reduce Overhead While Keeping Representative Traces

Why Use Probabilistic Sampling?

In large microservice environments, every request can generate hundreds of spans. Storing and transmitting all traces quickly overwhelms storage and network resources. Jaeger’s probability‑based sampler lets you keep only a representative subset of traces while discarding the rest before they even reach the collector.

Setting Up the Sampler

Agent‑Level Configuration

Configure the agent once, and all services that forward to it inherit the same sampling rate. The agent can be started with command‑line flags or a YAML config file.

# Command line
jaeger-agent \
  --sampling.type=probabilistic \
  --sampling.param=0.01 \
  --log-level=debug

# agent.yaml
sampling:
  type: probabilistic
  param: 0.01

Both snippets tell the agent to keep approximately 1% of root traces. The --log-level=debug flag is optional but useful for verification.

Environment Variable Override

When you prefer not to edit the agent’s binary flags, set:

export JAEGER_SAMPLER_TYPE=probabilistic
export JAEGER_SAMPLER_PARAM=0.01

These variables are read at startup; changing them requires a reload or restart.

Code‑Level Configuration (Go Example)

Below is a minimal Go snippet that creates a tracer with the same sampler settings. It mimics the agent’s behavior for local testing.

import (
    "github.com/uber/jaeger-client-go"
    "github.com/uber/jaeger-client-go/config"
)

conf := config.Configuration{
    Sampler: &config.SamplerConfig{
        Type:  config.SamplerTypeProbabilistic,
        Param: 0.01,
    },
    Reporter: &config.ReporterConfig{LogSpans: true},
}
tracer, closer, err := conf.NewTracer("my-service")
if err != nil {
    log.Fatalf("could not initialize tracer: %v", err)
}
defer closer.Close()

Remember: the sampler is applied *before* span creation, so dropping a root span automatically drops all its descendants.

How the Sampler Works

When a new trace starts, the sampler generates a random number between 0 and 1. If the number is less than the configured param, the trace is kept; otherwise it is dropped. The decision is stored in the trace context and propagated to downstream services. All services using the same sampler will honor the decision, ensuring consistency.

Verifying the Configuration

1. Start the agent with the debugging flag and generate 1,000 root spans (e.g., by sending 1,000 requests to a test endpoint).

2. Check the Jaeger UI or the collector’s trace count: you should see roughly 10 traces (1% of 1,000).

3. Search the agent logs for lines like Sampling decision: drop or Sampling decision: keep to confirm the sampler is invoked for each trace.

4. Optional: send a request with the HTTP header jaeger-sampling-decision=1 to force a keep decision. This header is respected only when the sampler type is remote; otherwise it is ignored.

Limitations & Common Mistakes

• No Guarantee for Specific Traces: The sampler is purely random; critical but rare traces may be missed.

• Per‑Trace, Not Per‑Span: The param applies to the root trace. Setting a high value (e.g., 0.5) will keep 50% of traces but still drop all child spans of a dropped root.

• Runtime Changes Require Reload: Updating JAEGER_SAMPLER_PARAM on the fly has no effect until the agent receives a SIGHUP or is restarted.

• Over‑Sampling Wastes Resources: Using a sampling rate close to 1.0 defeats the purpose and can overload the collector.

When to Use Probabilistic Sampling

Ideal for production environments where the volume of traces is high and you only need a statistical overview of system behavior. Use a lower rate (e.g., 0.01–0.05) for general monitoring, and raise the rate temporarily (e.g., to 0.5) when troubleshooting a specific issue.

Next Steps

Once you’re comfortable with the basic sampler, explore the remote sampler type, which delegates decisions to a central collector. This allows dynamic rate adjustments without restarting services.