Controlling Trace Volume with Jaeger’s Probabilistic Sampler: Design, Checks, and When to Re‑evaluate

Problem: Trace Flooding in High‑Traffic Environments

In large microservice deployments, every request can generate dozens of spans. If every span is stored, the collector, query, and storage layers quickly become saturated, leading to latency spikes or outages. Operators need a mechanism that limits trace volume while still providing a representative sample of traffic.

Requirement: A Simple, Configurable Volume Governor

• Apply a fixed sample rate to all incoming spans before they reach storage.
• Expose the effective rate to operators via metrics and UI.
• Operate within the trust boundary defined by the services that propagate trace context.
• Allow the rate to be changed without redeploying services.

Smallest Suitable Design: Probabilistic (Fixed) Sampler

Jaeger’s built‑in fixed‑rate sampler is the minimal component that satisfies the above. It is implemented on the collector side, so it does not require code changes in application services. The sampler drops spans that do not meet the probability threshold, preventing them from reaching the storage backend.

Configuration Example (Jaeger 1.45 JSON)

{
  "collector": {
    "sampler": {
      "type": "const",
      "param": 0.001
    }
  }
}

Running this JSON configuration in the Jaeger collector container (e.g., docker run -e "COLLECTOR_CONFIG=collector.json" jaegertracing/all-in-one:1.45) sets a 0.1 % sample rate. The collector will drop 99.9 % of incoming spans before they hit the storage backend.

YAML Syntax (Jaeger 1.40)

collector:
  sampler:
    type: const
    param: 0.001

Using the wrong format (JSON vs. YAML) silently ignores the sampler, so double‑check the file type and syntax.

Trust and Data Boundaries

The sampled trace set reflects the upstream services that propagate trace context. If a service does not inject a trace ID, its spans will never be sampled, creating a partial view of traffic. Therefore, the sampler’s effectiveness depends on the consistent propagation of context across the service mesh.

Operational Checks

1. UI Verification
   • Navigate to the Jaeger UI’s Sampling tab.
   • Confirm that the Sample Rate field matches the configured value.
   • Check the Sampled column in the trace list to ensure traces appear.
2. Metric Validation
   • Query Prometheus for the total number of sampled spans: sum(jaeger_sampler_sampled_total).
   • Compare the observed rate to the expected 0.001 probability using a short‑term average.
3. Synthetic Trace Test
   • Send a burst of 10,000 spans from a test client.
   • Expect roughly 10 stored spans (10,000 × 0.001).
   • Verify that the collector logs show the correct number of dropped spans.

Failure Modes

• Zero or Near‑Zero Rate – If the sampler is set to 0, no spans are stored, hiding failures. Verify with the UI that at least a few traces appear.
• Full Sampling (1.0) – A 100 % rate can overload storage, causing query timeouts. Monitor storage metrics and set an upper bound on the rate.
• Multiple Samplers – Defining both a collector and an agent sampler multiplies probabilities, unintentionally reducing coverage. Keep only one sampler in the chain.
• Mis‑formatting – JSON vs. YAML errors cause the sampler to be ignored. Validate the config file with a schema checker before deployment.

When to Re‑evaluate the Design

• When traffic patterns change significantly (e.g., a new service adds 10× the request rate). Re‑calculate the sample rate to keep storage usage within limits.
• When operators need higher fidelity for a specific service (e.g., debugging a fault). Temporarily bump the rate or apply a per‑service sampler.
• When storage costs become a concern. Lower the rate or switch to a more aggressive sampling strategy (e.g., rate‑based).
• When the Jaeger version upgrades and deprecates the current sampler syntax. Update the config accordingly.

Practical Check: Confirming Effective Sampling

After deploying the sampler, run the following Prometheus query to verify the rate:

rate(jaeger_sampler_sampled_total[1m])

If the result is close to the configured probability (within ±10 %), the sampler is functioning. If the value is significantly lower, investigate missing trace context propagation or mis‑configured sampler.

Conclusion

Jaeger’s probabilistic sampler offers a lightweight, declarative way to control trace volume. By configuring a fixed sample rate, verifying it through the UI and Prometheus metrics, and monitoring for failure modes, operators can keep tracing systems performant while still gaining insight into distributed traffic patterns.