Choosing Between Head-Based and Tail-Based Sampling in OpenTelemetry Collector

Decision: When to Apply Sampling in a Distributed Tracing Pipeline

Distributed tracing creates high volume data. The engineering decision is whether to make the sampling choice at trace start (head‑based) or after the trace completes (tail‑based). This guide outlines the constraints, compares the supported options in a compact table, explains the trade‑offs, and shows a concrete Collector configuration for a hybrid approach.

Constraints and Assumptions

• Throughput: Head‑based adds negligible CPU/memory; tail‑based requires the Collector to buffer spans until a decision can be made.
• Signal fidelity: Head‑based cannot see latency or errors that occur later in the trace; tail‑based can.
• Where it runs: Head‑based runs in language SDKs or the Collector’s probabilistic_sampler processor. Tail‑based runs only in the Collector via the tailsamplingprocessor (contrib distribution).
• Propagation: Both respect the W3C TraceContext traceparent sampled flag; a trace dropped at the head never reaches tail policies.
• Version assumption: The configuration examples assume OpenTelemetry Collector Contrib version >=0.84.0, where the tailsamplingprocessor is stable. Adjust field names if you use an older release.

Supported Options at a Glance

Dimension	Head‑Based	Tail‑Based
Where it runs	Language SDK or Collector probabilistic_sampler	Collector only (contrib distribution)
Decision point	Trace start (first span)	After trace completes or decision_wait expires
Policy types	Trace‑ID ratio, parent‑based, always‑on/off	Attribute match, latency, status code, rate limiting
Resource cost	Negligible CPU/memory	Memory proportional to buffered traces
Can sample on errors/latency	No	Yes
Configuration change	SDK restart or dynamic config	Collector restart or reload

Trade‑Offs in Practice

Head‑Based: Predictable, Low‑Cost, Blind to Outcomes

Use head‑based when you need a fixed percentage of all traffic (e.g., 5 %) and can accept missing rare errors that fall outside the sample. Configure it per service or centrally in the Collector:

# Java SDK example
Sampler sampler = Sampler.traceIdRatioBased(0.05); // 5%
SdkTracerProvider provider = SdkTracerProvider.builder()
    .setSampler(sampler)
    .build();

Or in the Collector with the probabilistic_sampler processor before export. Trace‑ID ratio sampling is deterministic: the same trace ID always yields the same decision, so parent‑based sampling propagates consistently across service boundaries.

Tail‑Based: Signal‑Aware, Resource‑Heavy

Tail‑based shines when you must capture every error trace or every request exceeding a latency SLO. The Collector buffers spans grouped by trace ID until the trace ends or decision_wait fires, then evaluates policies over the complete span set. Common policy types include:

• string_attribute — match an attribute such as http.status_code
• latency — threshold in milliseconds
• status_code — match ERROR status
• rate_limiting — cap traces per second after other policies

The catch: you must size the trace buffer (num_traces) and expected throughput for your peak load. Undersizing causes spans to be dropped before a decision; oversizing wastes memory. The tail sampling processor exposes metrics (names vary by release—check the /metrics endpoint on your Collector) covering sampled/not‑sampled trace counts and dropped spans. Alert on dropped spans to stay ahead of out‑of‑memory risk.

Recommended Hybrid: Head Filter → Tail Policy

For high‑throughput systems, chain a lightweight probabilistic sampler in the Collector to reduce volume before the tail sampling processor. This keeps tail‑based memory bounded while preserving error and latency signal on the surviving traffic.

Collector Configuration Example

Save as collector-config.yaml. Requires a Collector distribution that includes the contrib processors. Run with otelcol-contrib --config=collector-config.yaml as a user with read access to the config file and network access to the exporter endpoint.

receivers:
  otlp:
    protocols:
      grpc:
      http:

processors:
  # Head‑based: keep ~5% of all traces
  probabilistic_sampler:
    sampling_percentage: 5

  # Tail‑based: evaluate remaining traces
  tail_sampling:
    decision_wait: 30s
    num_traces: 50000
    expected_new_traces_per_sec: 1000
    policies:
      - name: errors
        type: string_attribute
        string_attribute:
          key: http.status_code
          values: ["500", "501", "502", "503", "504"]
      - name: slow
        type: latency
        latency:
          threshold_ms: 500
      - name: baseline
        type: rate_limiting
        rate_limiting:
          spans_per_second: 100

  batch:
    timeout: 10s

exporters:
  otlp:
    endpoint: "tempo:4317"
    tls:
      insecure: true

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [probabilistic_sampler, tail_sampling, batch]
      exporters: [otlp]

What This Config Does

1. probabilistic_sampler keeps about 5 % of traces at ingest; the rest are dropped immediately.
2. The remaining traces enter tail_sampling. Each trace is held up to decision_wait: 30s; set this above your p99 trace duration.
3. Policies run in order: errors catches HTTP 5xx; slow catches traces exceeding 500 ms; baseline rate‑limits the rest.
4. batch aggregates spans before export.

Validation Steps

Run the Collector locally and send test traffic using an OTLP‑capable load generator or instrumented app. Because CLI tooling flags change between versions, the steps below describe intent.

1. Start the Collector

# Terminal 1
 otelcol-contrib --config=collector-config.yaml

2. Generate Test Traces

• A trace with http.status_code=500 — should be sampled by the errors policy.
• A trace whose root span duration exceeds 500 ms — should be sampled by the slow policy.
• Normal, fast, successful traces — mostly dropped by the 5 % head sampler.

3. Check Collector Metrics

# Default Prometheus endpoint
curl -s localhost:8888/metrics | grep -i tail_sampling

Look for counters covering sampled and dropped traces. Exact metric names differ across versions; grep broadly and confirm against your release documentation.

4. Verify Exported Spans

Query your backend for the test service. Confirm error and slow traces appear and carry the W3C sampled flag (traceparent ending in -01).

Limitations and Gotchas

• No retroactive sampling: A trace dropped by the 5 % head sampler never reaches tail policies. If you must capture all errors, remove the head sampler or use an SDK‑level sampler that always samples on error.
• decision_wait too short: If your p99 trace duration is 45 s but decision_wait is 30 s, incomplete traces are evaluated early and may miss the signal.
• Memory pressure: Buffer size is roughly num_traces × average spans per trace × average span size. Measure your actual span sizes rather than trusting generic estimates.
• Config changes: Changing sampling rates requires a Collector restart or a reload mechanism (e.g., OpenTelemetry Operator).
• Double head sampling: If services already run SDK‑level samplers, the Collector's probabilistic sampler sees only the already‑sampled subset.

Quick Decision Checklist

If…	Start With
You need a fixed % of all traffic, low overhead	Head‑based (SDK or Collector)
You must capture every 5xx / latency breach	Tail‑based (tail sampling processor)
High throughput + must‑capture errors	Hybrid: light head sampler → tail processor
Team cannot operate a stateful Collector tier	Head‑based only; custom SDK sampler