Which Jaeger Sampling Strategy Best Supports Accurate Bottleneck Detection?

Recommended Strategy: Tail-Based Sampling

To accurately detect latency bottlenecks under high-throughput constraints, tail-based sampling is the most effective strategy. Unlike head-based sampling, which decides to keep a trace at the start of the request, tail-based sampling makes the decision after the entire trace has completed. This allows you to define policies that specifically retain traces with high latency or error statuses, ensuring that the "outlier" traces—which typically contain the bottleneck evidence—are captured even if the overall sampling rate is low (e.g., 1%).

Probabilistic vs. Tail-Based Logic

The following table outlines why adaptive or probabilistic head-based sampling often fails to expose subtle bottlenecks:

Strategy	Decision Point	Bottleneck Visibility
Head-Based (Probabilistic)	Root Span Start	Low: High-latency spikes are sampled randomly and often missed.
Adaptive Sampling	Dynamic/Collector Load	Moderate: Prevents overload but still lacks latency-awareness.
Tail-Based Sampling	Trace Completion	High: Specifically retains traces exceeding a latency threshold.

Handling Clock Skew and UI Fidelity

Regarding your constraints on clock skew and UI visualization:

• Clock Skew (±50ms): The Jaeger UI calculates span durations based on the start_time and end_time of individual spans. Because these are recorded by the same local clock, the duration of a single span remains accurate. However, the relative offset between spans from different services will be shifted. A 50ms skew can make a child span appear to start before its parent, or create artificial gaps in the timeline.
• Fidelity: If you use 1% head-based sampling, you are statistically likely to miss short-duration spikes unless they occur with extreme frequency. Tail-based sampling solves this by treating "latency > X ms" as a trigger for 100% retention of that specific trace.

Implementation Steps for Bottleneck Detection

1. Deploy a Sampling Proxy: Implement a tail-based sampler (such as the OpenTelemetry Collector with the tailsamplingprocessor) between your services and the Jaeger collector.
2. Define Latency Policies: Configure the sampler to retain 100% of traces where the root span duration exceeds your defined "bottleneck" threshold (e.g., 500ms).
3. Set a Baseline Rate: Maintain a low probabilistic rate (e.g., 0.1% to 1%) for healthy traces to provide a performance baseline for comparison.
4. Verify with Synthetic Spikes: Inject a known latency (e.g., a 500ms sleep) into a downstream service and verify that the resulting trace is captured despite the low overall sampling rate.

Missing Diagnostic Detail: Are you using the OpenTelemetry Collector as a gateway, or are services sending spans directly to the Jaeger Collector? (Tail-based sampling requires a stateful buffer/proxy to hold spans until the trace completes; it cannot be performed by the Jaeger Collector alone in a stateless manner).