Head-Based SDK Sampling vs. Collector Tail Sampling: Where Should the Keep/Drop Decision Live?
0 reputation · 30 Apr 2022, 04:51 UTC
I'm designing the tracing pipeline for a set of services that will export OpenTelemetry spans through a Collector, and I need to pick a sampling strategy before rollout. The two documented options pull in opposite directions.
With head-based sampling (for example a ParentBased sampler wrapping TraceIdRatioBased in the SDK), the decision happens at span creation, so dropped traces never hit the network or storage. The cost is that the decision is blind: a trace that later turns out to contain an error may already be gone, and there is no way to recover it.
With tail-based sampling in the Collector's tail_sampling processor, I could keep only traces matching policies like "contains an error" or "exceeds a latency threshold." But this requires buffering whole traces in memory, a decision wait window, and trace-ID-aware routing so all spans of a trace reach the same Collector replica.
Assume current stable SDK and Collector releases; exact sampler and processor configuration is version-sensitive.
Given a multi-replica Collector deployment, which trade-off is more defensible: cheap but blind head sampling, or informed but stateful tail sampling? If I combine a low head rate with tail policies, how do I reason about effective sampling probability and trace completeness? What breaks operationally if the tail buffer is undersized under load?