Choosing OpenTelemetry Sampling: Head-Based vs Tail-Based

Deciding How OpenTelemetry Samples Traces

If you instrument multiple services with OpenTelemetry, every trace generates spans, storage costs, and query load. The decision is where to apply sampling: at the application SDK before spans are exported, or later in the OpenTelemetry Collector after spans from several services arrive. The useful takeaway is to start with head-based sampling for simple services, and move to tail-based sampling in the Collector only when you need to keep traces that contain errors while discarding traces from healthy requests.

Sampling Options Comparison

Option	Where it runs	Decision basis	Best for
Head-based: always_on	Application SDK	Every trace is sampled	Critical paths, low-volume services
Head-based: probabilistic	Application SDK	Fixed percentage of traces	High-volume services where cost matters more than completeness
Head-based: always_off	Application SDK	No traces are sampled	Disabling tracing in specific environments
Tail-based	OpenTelemetry Collector	Complete trace after all spans arrive	Keeping error traces, dropping successful ones, service-level inspection

Trade-offs to Consider

Head-based sampling decides whether to keep a trace before the application knows whether the request will fail. It is cheap and requires no extra infrastructure, but it cannot preserve only error traces. Tail-based sampling waits until all spans for a trace reach the Collector, so it can make smarter decisions, but it holds partial traces in memory and requires every span of a trace to reach the same Collector instance or a Collector group that shares state.

Resource overhead depends heavily on sampling rate and tail buffer size. A tail-based Collector consumes more memory than a simple exporter because it must store incomplete traces until they time out or complete.

Concrete Implementation: Tail-Based Sampling in the Collector

This example configures the OpenTelemetry Collector to sample traces based on whether they contain an error status code, with a fallback probabilistic rate for healthy traffic.

Where to run: the OpenTelemetry Collector configuration file, typically otel-collector-config.yaml.

Permissions: read access to the configuration file and permission to restart the Collector process.

Placeholders: replace TRACE_BACKEND_ENDPOINT with your backend OTLP endpoint.

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch:
  tailsampling:
    policies:
      - name: keep-errors
        type: status_code
        status_code:
          status_codes:
            - ERROR
      - name: sample-healthy
        type: probabilistic
        sampling_percentage: 5

exporters:
  otlp:
    endpoint: "TRACE_BACKEND_ENDPOINT:4317"

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch, tailsampling]
      exporters: [otlp]

The tailsampling processor is part of the OpenTelemetry Collector contrib distribution. Head-based sampling is configured in the application SDK instead, for example by setting a sampler on the tracer provider before creating spans.

Expected check: run otelcol --config otel-collector-config.yaml and confirm the process starts without configuration errors. The Collector should accept OTLP spans and export only those matching a sampling policy.

Risk: if spans from the same trace are sent to different Collector instances that do not share tail-sampling state, one instance may keep a trace while another drops it, producing incomplete traces in the backend.

Validation Steps

1. Instrument a service with the OpenTelemetry SDK and export traces to the Collector configured above.
2. Send a request that completes successfully and a request that returns a server error.
3. Query your trace backend for both trace IDs.
4. Confirm the error trace is retained and the successful trace is either retained at the configured probabilistic rate or dropped.

Limitation and How to Check the Result

Tail-based sampling increases Collector memory usage because incomplete traces are buffered. Verify the result by checking the Collector's internal metrics, such as the number of spans in the tail sampling buffer, and by confirming in your backend UI that sampled traces contain the expected spans. If you see partial traces, review Collector routing or switch to head-based sampling for that service.

Version assumption: this guide refers to OpenTelemetry Collector releases that include the tailsampling processor and SDKs that support standard samplers. Check your SDK and Collector versions before deploying.