Head-Based SDK Sampling vs. Collector Tail Sampling: Where Should the Keep/Drop Decision Live?

The most defensible trade-off depends on your primary goal: cost and performance predictability (Head-based) or high-fidelity error capture (Tail-based). For most high-traffic production environments, a hybrid approach is the most sustainable, but it requires a specific architectural prerequisite: trace-ID-aware routing.

The Trade-off Analysis

Head-Based Sampling (Cheap but Blind)

Decisions are made at the start of the trace. This is the most stable operational choice because it eliminates network and memory overhead for dropped traces before they ever leave the application process.

• Defensibility: High for systems where cost-per-span is the primary constraint and general trends are more important than individual edge-case errors.
• Risk: You will lose "needle in a haystack" events. If a trace is sampled out at the head, no amount of downstream logic can recover it.

Tail-Based Sampling (Informed but Stateful)

Decisions are made after all spans for a trace have been collected. This allows you to keep 100% of errors and high-latency traces while dropping 99% of healthy ones.

• Defensibility: High for debugging complex distributed systems where errors are rare but critical to diagnose.
• Risk: Significant operational complexity. In a multi-replica Collector setup, you must use a load balancer (like an OpenTelemetry Collector Gateway) that routes spans with the same TraceID to the same replica. Without this, the tail sampler sees only fragments of a trace and will make incorrect drop decisions.

Reasoning About Hybrid Sampling

When combining a low head-sampling rate (e.g., 10%) with tail policies, your effective sampling probability is the product of the two. If your head sampler is set to 0.1 and your tail policy keeps all errors, you are not keeping all errors—you are keeping 10% of all errors.

The Logic: Head sampling acts as a "pre-filter" to protect the Collector's memory. Tail sampling then acts as a "refiner" to ensure the remaining budget is spent on the most valuable traces.

Operational Failure Modes

If the tail buffer is undersized under heavy load, the Collector typically handles overflow by dropping the oldest traces in the buffer to make room for new ones. This leads to trace fragmentation: you may see the end of a trace (the error) but lose the beginning (the root cause), rendering the trace nearly useless for debugging.

Verification Steps

1. Calculate Buffer Needs: Estimate (Traces per second) × (Average spans per trace) × (Average span size) × (Decision window in seconds).
2. Monitor Occupancy: Track the otelcol_processor_tail_sampling_buffer_occupancy metric. If it consistently hits the limit, increase memory or tighten head-based sampling.
3. Validate Routing: Send a single trace across multiple services and verify that all spans arrive at a single Collector replica before the tail decision is made.

Missing Diagnostic: To provide a specific memory recommendation, what is your average span count per trace and your target decision window (e.g., 30 seconds)?