Choosing Between Head-Based and Tail-Based Trace Sampling in OpenTelemetry

Decision and Constraints

When operating a distributed service that generates millions of spans per day, you must decide where to apply trace sampling. The goal is to stay within a limited ingest budget while preserving enough visibility to debug errors and latency outliers. The decision hinges on three constraints:

• Ingest budget: Backend storage and processing can only handle a fixed number of traces per second.
• Edge latency: Sampling decisions made at the service instance should add minimal overhead.
• Signal quality: Traces that represent errors or unusually high latency must be retained with high probability.

These constraints lead to a choice between head‑based sampling (decided in the SDK) and tail‑based sampling (decided in the OpenTelemetry Collector after a trace is complete).

Options Comparison

Option	Where decided	Cost	Completeness	Typical use
Head‑based probabilistic (TraceIdRatioBasedSampler)	SDK	Low – constant‑time decision, no extra memory	Uncorrelated; early spans may be dropped, losing context	High‑volume steady‑state traffic where uniform sampling is acceptable
Head‑based rate limiting (Sampler that caps spans per second)	SDK	Low – simple counter, minimal CPU	Local cap only; no global view of trace budget	Burst protection at the edge when you need a hard limit per instance
Tail‑based policy (tail sampling processor)	Collector	Higher – requires buffering traces in memory until completion	Full trace view; can keep spans based on error status, duration, or attributes	Cost control while preserving anomalies and debugging signals

Trade‑offs

• Head‑based: Predictable load, no coordination between services, minimal SDK overhead. However, it cannot enforce a global ingest budget and may discard rare error traces before they are seen.
• Tail‑based: Improves signal quality because the decision is made with the complete trace context, enabling policies that retain errors or long‑latency spans. The trade‑off is increased collector memory and processing, added tail latency for export, and dependence on collector version and vendor support.
• Hybrid approach: Many teams apply a loose head‑based sampler (e.g., 10 %) to reduce the volume sent to the collector, then use tail‑based sampling on that subset to refine the final decision. This balances SDK simplicity with collector‑level signal control.

Concrete Implementation

Head‑based configuration (Java SDK)

Configure a ParentBasedSampler that wraps a TraceIdRatioBasedSampler set to 1 % sampling. This ensures that child spans inherit the sampling decision of their parent.

import io.opentelemetry.sdk.trace.sampling.*;

Sampler sampler = SamplerParentBased.create(
    SamplerTraceIdRatioBased.create(0.01)   // 1 % probability
);

 // When building the OpenTelemetry SDK:
SdkTracerProvider.builder()
    .setSampler(sampler)
    .build();

Tail‑based configuration (OpenTelemetry Collector)

Enable the tail_sampling processor and define a policy that retains spans with an error status or a duration greater than 500 ms.

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch, tail_sampling]
      exporters: [logging]

processors:
  tail_sampling:
    decision_wait: 30s
    num_traces: 50000
    expected_new_traces_per_sec: 5000
    policies:
      - name: error-policy
        type: status
        status_code: ERROR
      - name: latency-policy
        type: latency
        latency: {threshold_ms: 500}

Validation steps

1. Start a local Collector with the tail‑sampling config above.
2. Instrument a test service with the Java SDK head‑based sampler (1 %).
3. Emit synthetic spans:
   • One span with status ERROR.
   • One span with duration 600 ms.
   • Several normal spans (OK, 200 ms).
4. Check the Collector logs for tail_sampling hits on the error and latency policies.
5. Inspect the exported traces (e.g., via the logging exporter) and verify that the sampling flag (sampled) is true for the retained spans and false for dropped ones.
6. Measure the ingest volume over a minute and compare it to the expected rate based on the head‑based rate (1 %) and the tail‑sampling retention ratio.

Limitations and Practical Checks

Tail sampling is not uniformly mature across all Collector distributions; verify that your Collector version includes the tail_sampling processor (see release notes). The sampler API in the SDK changed between OpenTelemetry 1.0 and 1.12, so consult the SDK documentation for your language version.

Because sampling decisions affect trace completeness, any metrics derived from traces (e.g., error rates computed from span counts) will be biased unless you adjust for the sampling probability. To check the result in practice:

• Enable the sampling attribute on exported spans and compute the ratio of sampled=true spans to total generated spans in a short test window.
• Compare that ratio to the configured head‑based probability (or the effective retention after tail‑sampling) to confirm the sampler is behaving as expected.
• Run a load test in a staging environment that mirrors production traffic, record the ingest volume, and verify it stays within your budget while still capturing a statistically significant number of error and high‑latency traces.

If the observed retention deviates significantly, revisit the Collector’s num_traces and expected_new_traces_per_sec settings, or consider adjusting the head‑based rate to reduce the load on the tail‑sampling processor.