Solving the 'Missing Error' Problem with Jaeger Tail-Based Sampling

The Blind Spot of Head-Based Sampling

Most distributed tracing setups start with head-based sampling. The decision to keep a trace is made at the very first span (the 'head'), usually based on a fixed percentage like 1%. While this keeps storage costs low, it creates a critical visibility gap: you are just as likely to drop a trace that resulted in a 500 Internal Server Error or a 10-second timeout as you are to drop a successful 20ms request.

When an incident occurs, you often find that the exact trace you need to debug the outage was sampled out. To fix this, you need tail-based sampling. Instead of deciding at the start, the system buffers all spans for a trace and decides whether to keep it only after the entire trace has completed or a timeout is reached.

How Tail-Based Sampling Works

Tail-based sampling moves the decision logic from the application SDK to the Jaeger Collector (or an OpenTelemetry Collector). The process follows a specific sequence:

• Buffering: As spans arrive, the Collector groups them by Trace ID in an in-memory buffer.
• Windowing: The Collector waits for a configurable window (e.g., 30 seconds) to ensure most spans for that trace have arrived.
• Policy Evaluation: Once the window closes, the Collector evaluates the trace against a set of rules. If any rule matches, the entire trace is forwarded to storage.
• Eviction: Traces that don't match any "keep" policy are dropped from memory without ever hitting the database.

Implementing a High-Signal Sampling Policy

The goal is to capture 100% of the "interesting" traces (errors and outliers) while keeping only a tiny fraction of the "boring" ones (successful, fast requests). This is achieved through a prioritized list of policies.

Below is a conceptual configuration for a Collector using the tailsamplingprocessor logic. This setup ensures that any trace with an error or high latency is preserved, while only 1% of healthy traffic is kept for baseline metrics.

# Example Tail Sampling Policy Configuration
processors:
  tail_sampling:
    # Wait 30s for all spans to arrive before deciding
    decision_wait: 30s
    # Maximum number of traces to buffer in memory
    num_traces: 10000
    policies:
      - name: keep-errors
        type: status_code
        status_code: {status_code: ERROR}
      - name: keep-slow-requests
        type: latency
        latency: {threshold_ms: 2000}
      - name: baseline-traffic
        type: probabilistic
        probabilistic: {sampling_percentage: 1}

Execution Details

• Where to run: This configuration is applied to the Jaeger Collector or the OpenTelemetry Collector.
• Permissions: The process requires sufficient heap memory to maintain the buffer.
• Expected Result: Your storage backend will show a high concentration of failed requests and p99 latency outliers, with a sparse sampling of successful requests.

Operational Trade-offs and Constraints

Tail-based sampling is not a "free" upgrade; it introduces statefulness to your telemetry pipeline.

The Memory Cost

Unlike head-based sampling, which is stateless, tail sampling requires RAM to hold spans. Memory usage scales by: Ingest Rate × Buffer Window × Average Spans per Trace. For a workload of 10k spans/sec with a 30s window, you should allocate several gigabytes of heap to avoid buffer overflows.

The Scaling Challenge (Trace Affinity)

Because the decision happens in the Collector's memory, all spans for a single Trace ID must land on the same Collector instance. If you have multiple Collector replicas behind a round-robin load balancer, spans will be split across instances, and policies will be evaluated against partial traces, leading to incorrect sampling decisions.

To solve this, you must implement Trace ID affinity (sticky routing) at the load balancer level or use a load-balancing layer like Kafka to ensure specific Trace IDs are always routed to the same consumer.

Verification and Monitoring

To verify your sampling is working, monitor the following Prometheus metrics on your Collector:

• jaeger_collector_tail_sampling_buffer_spans: Ensures your buffer isn't constantly hitting its limit.
• jaeger_collector_tail_sampling_traces_dropped: Tracks how many traces were evicted.

A practical test is to trigger a known error (e.g., a 500 error in a staging environment) and verify that the trace appears in the Jaeger UI 100% of the time, regardless of your probabilistic baseline rate.