Taming Trace Volume with Jaeger Adaptive Sampling

The Cost of Visibility

Distributed tracing is essential for debugging microservices, but it introduces a classic engineering trade-off: visibility versus overhead. If you sample 100% of requests, your storage costs skyrocket and your network becomes saturated during traffic spikes. If you use a static 1% sampling rate, you lose the rare, intermittent errors that only happen once every ten thousand requests, and you get zero visibility into low-traffic services.

The goal is to maintain a consistent Traces Per Second (TPS) budget. You want enough data to be statistically significant for high-traffic endpoints, but you don't want to miss the heartbeat of your quietest services. This is where Jaeger's adaptive sampling strategy becomes critical.

How Adaptive Sampling Works

Unlike static probabilistic sampling—where a fixed percentage of traces are kept regardless of volume—adaptive sampling dynamically adjusts the sampling probability based on actual traffic patterns. It operates on a feedback loop between the Jaeger Collector and the Jaeger Agent.

The Collector monitors the incoming trace volume for every unique service and operation pair. It uses a token-bucket style controller to track the current TPS. If an operation exceeds its allocated budget, the Collector lowers the sampling probability for that specific operation. If traffic drops, it raises the probability to ensure you still have enough data for analysis.

To avoid making a network call for every single span, the Collector pushes these updated probabilities to the Jaeger Agents. The Agents cache these strategies (typically refreshing every 10 seconds), allowing the instrumentation point to make sampling decisions locally with minimal latency.

Implementing a Sampling Strategy

To enable adaptive sampling, you must provide the Jaeger Collector with a strategy file. This file defines the global budget and allows for granular overrides for specific services that require higher visibility.

Example Strategy Configuration

Create a JSON file (e.g., sampling-strategies.json) to define your targets. In this example, we set a global default but ensure our critical payment-gateway service always has a higher budget.

{
  "default_strategy": {
    "type": "adaptive",
    "max_traces_per_second": 5
  },
  "per_service_strategies": {
    "payment-gateway": {
      "max_traces_per_second": 20
    },
    "auth-service": {
      "max_traces_per_second": 10
    }
  }
}

Deployment and Activation

Run the Jaeger Collector with the --sampling.strategies-file flag. You will need permissions to read the file on the host or within the container volume.

# Run the collector with the strategy file
./jaeger-collector --sampling.strategies-file=/etc/jaeger/sampling-strategies.json

Pro Tip: You do not need to restart the Collector to update these limits. If you modify the JSON file, send a SIGHUP signal to the Collector process to trigger a hot reload of the strategy.

Trade-offs and Operational Limits

Adaptive sampling is powerful, but it introduces specific architectural dependencies and risks:

• Agent Dependency: This strategy relies on the Collector-Agent communication channel. If network partitions occur, Agents will rely on stale cached strategies, leading to "sampling drift" where the actual TPS deviates from your budget.
• Cardinality Risk: The Collector maintains a bucket for every unique operation. If your service uses high-cardinality operation names (e.g., including a UserID in the operation name), the Collector's memory usage will climb linearly. Always use generic operation names like GET /user/{id} rather than GET /user/12345.
• Convergence Lag: The system is reactive, not predictive. During a sudden, massive traffic spike, you may see a brief surge in stored traces before the controller converges on a lower probability.

Verifying the Result

To confirm the adaptive sampler is working, you can use the Collector's internal metrics. Monitor the jaeger_agent_sampling_strategy_fetch_total Prometheus metric to ensure Agents are successfully pulling updates.

For a manual check, generate a burst of traffic to a specific endpoint and observe the Sampling tab in the Jaeger UI. You should see the sampling probability decrease as the TPS hits the max_traces_per_second limit defined in your configuration file.