Implementing Adaptive Sampling in Jaeger to Control Trace Volume

The Challenge of Static Sampling

In a large microservices architecture, static sampling—where every service is told to keep 1% or 10% of traces—creates a visibility gap. High‑traffic endpoints (like /healthz) flood your storage with redundant data, while critical but rare error paths in low‑traffic services are often dropped entirely. Manually tuning these rates across hundreds of services is operationally impossible.

The solution is Adaptive Sampling. Instead of static percentages, the Jaeger Collector monitors the actual arrival rate of spans and dynamically pushes updated sampling strategies back to the clients. This ensures you capture a consistent number of traces per operation regardless of traffic volume.

Prerequisites

• Jaeger Collector: Version 1.20+ (recommended for stable remote sampling).
• Compatible SDKs: Client libraries that support remote sampling (e.g., Jaeger Java, Go, or OpenTelemetry with the Jaeger exporter).
• Network Connectivity: Services must be able to reach the Collector’s sampling endpoint to receive strategy updates.

Configuring the Adaptive Sampling Strategy

Adaptive sampling moves the decision‑making logic from the application code to the Collector. The Collector tracks the number of spans per operation and calculates a sampling rate that keeps the volume within a target range.

1. Collector Configuration

To enable adaptive sampling, you must configure the Collector to manage sampling strategies. If using the Jaeger Operator or Helm chart, you need to define the sampling strategy store. While in‑memory storage is available for testing, a persistent store is required for production to avoid resetting sampling rates on every restart.

# Example Collector flags for adaptive sampling
--collector.sampling.strategy-store=memory 
--collector.sampling.target-spans-per-second=10

The target-spans-per-second flag is the primary lever. It tells the collector: "I want roughly 10 traces per second for any given operation, regardless of whether that operation is called 100 times or 100,000 times per second."

2. Client‑Side Integration

The application does not need a hardcoded sampling rate. Instead, it must be configured to poll the Collector for its sampling strategy. In a Java Spring Boot environment using the Jaeger client, this is typically handled by setting the jaeger.sampler.manager.host property.

3. Deployment Execution

1. Update Collector: Apply the target-spans-per-second configuration to the Jaeger Collector deployment.
2. Restart Services: Restart your microservices to ensure they initialize the remote sampling manager and connect to the Collector.
3. Observation Period: Allow the system to run for 15–30 minutes. The Collector needs a baseline of traffic to calculate the initial sampling rates.

Comparing Static vs. Adaptive Behavior

Verification and Diagnostics

Because adaptive sampling happens in the background, you must verify it via logs and metrics rather than application code.

Check Collector Logs

Run the following command on your Kubernetes cluster to see if the Collector is pushing updates:

# Replace jaeger-collector with your actual pod name
kubectl logs -l app=jaeger-collector | grep \"sampling strategy\"

Expected Result: You should see entries indicating that sampling strategies are being calculated and updated for specific operation names.

Cross‑Reference with Application Metrics

Compare the Total Request Count (from Prometheus/Micrometer) against the Spans Received (in the Jaeger UI). If an endpoint has 1,000 requests/sec and your target is 10/sec, the Jaeger UI should show approximately 1% of those traces, while a low‑traffic endpoint (1 request/sec) should show nearly 100% capture.

Limitations and Risks

• The "Cold Start" Problem: When a brand new operation is deployed, the Collector has no history. It will initially use a default rate, which may lead to a brief spike in traces or a gap in visibility until the first feedback loop completes.
• Sampling Lag: There is a delay between a traffic spike and the Collector pushing a lower sampling rate. During this window, your storage may experience a temporary surge.
• SDK Compatibility: If using OpenTelemetry, ensure the OTel Collector is configured to translate Jaeger's remote sampling strategies, as the protocols differ.

Rollback Procedure

If adaptive sampling causes instability or excessive data loss, revert to static sampling by:

1. Removing the --collector.sampling.strategy-store flag from the Collector.
2. Updating the application environment variables to use a constant sampler (e.g., JAEGER_SAMPLER_TYPE=const and JAEGER_SAMPLER_PARAM=1).
3. Restarting the services to force them to stop polling the Collector.