Taming Trace Volume: Implementing Adaptive Sampling in Jaeger

The Storage Cost of Full Observability

In a high‑traffic distributed system, you face a constant trade‑off: sample too little, and you miss the rare, intermittent bug that only happens once every 10,000 requests; sample too much, and your storage costs explode or your tracing backend crashes during a traffic spike. Constant sampling rates are a blunt instrument that fails during volatility.

The solution is Adaptive Sampling. Instead of a hard‑coded percentage, Adaptive Sampling lets the Jaeger Collector dynamically adjust sampling probabilities based on real‑time traffic volume. The goal is to maintain a consistent number of samples per second, regardless of whether your system is idling or under a 10× load surge.

How Adaptive Sampling Functions

Adaptive Sampling shifts the decision‑making process from the client (the application) to the Collector. In a standard setup, the client decides whether to sample a trace based on a static configuration. With adaptive sampling, the Collector monitors the throughput of specific operations and calculates the probability required to hit a target sample rate.

The process follows a feedback loop:

• Observation: The Collector tracks the number of spans arriving for a specific operation.
• Calculation: It compares the current throughput against a pre‑defined target (e.g., 10 traces per second).
• Propagation: The Collector updates the sampling strategy, which is then propagated back to the clients.
• Execution: The clients apply the new probability to incoming requests.

Practical Configuration Example

To implement this, you must configure the Jaeger Collector to manage the sampling strategies. While specific configurations vary by deployment (Operator vs. Helm), the logic centers on defining the target throughput for your operations.

Consider a scenario where you have a /checkout endpoint. You want to ensure you see enough data to debug latency but don’t want to store 5,000 traces per second during a flash sale.

# Example conceptual configuration for the Collector
# Target: Maintain 5 traces per second for the checkout operation
# Max: Never exceed 100 traces per second regardless of target
# Min: Always sample at least 0.1% to ensure some visibility

sampling_strategies:
  - operation: \"checkout\"
    target_tps: 5
    max_tps: 100
    min_probability: 0.001

Verification Steps:
To verify this is working, run these checks on your Jaeger instance:

1. Monitor Admin Metrics: Access the Collector’s admin port to observe the current calculated sampling probabilities.
2. Simulate Load: Use a tool like k6 or Apache Benchmark to send a surge of requests to the target operation.
3. Observe the Shift: Check the Jaeger UI or logs. You should see the sampling probability decrease as the request volume increases, keeping the total stored spans relatively flat.

The Trade‑offs of Dynamic Rates

Adaptive sampling is not a silver bullet. It introduces two primary technical risks:

The Long‑Tail Blind Spot

When traffic spikes, the sampling probability drops. If a critical error occurs only in 0.01% of requests during that peak, and your adaptive rate has dropped to 0.1%, the mathematical likelihood of capturing that specific error decreases significantly. You are trading the “long‑tail” of rare events for system stability.

Propagation Lag

There is a temporal gap between the moment the Collector detects a spike and the moment the Client receives the updated sampling rate. During a sudden, vertical spike in traffic, your backend may still experience a brief surge of spans before the adaptive rate kicks in and throttles the volume.

Closing Strategy

Adaptive sampling is the right choice when your traffic patterns are unpredictable and your storage budget is finite. To mitigate the risk of missing rare errors, combine adaptive sampling with tail‑based sampling (where decisions are made after the trace is complete, allowing you to keep all traces that resulted in an error regardless of the rate). Start by setting conservative min_probability values to ensure you never go completely blind to any single service operation.