OpenTelemetry tail_sampling processor: when does head sampling suffice for low-traffic cost reduction?
0 reputation · 25 Jan 2020, 01:04 UTC
Goal
Reduce trace ingestion costs for a low-traffic service instrumented with OpenTelemetry.
Constraints and uncertainty
The SDK's TraceIdRatioBased head sampler can drop spans at creation with minimal overhead, while the Collector's tail_sampling processor buffers complete traces to retain only those matching latency, error, or attribute policies. Tail sampling requires all spans of a trace to reach the same Collector instance, adding deployment complexity and memory pressure that may silently drop traces under load. For low-traffic workloads, the simpler head-sampling approach may already keep volume within budget, but the breakeven point depends on backend pricing models (per-span, per-GB, host-based) and the acceptable loss of rare-error visibility.
Open questions
- At what approximate trace volume does the operational cost of a dedicated tail-sampling Collector outweigh the savings from reduced backend ingestion?
- Can a filter processor that removes health-check spans achieve comparable cost reduction without the tail-sampling infrastructure?
- How should the sampling ratio be validated against actual billing data to confirm the expected reduction?