Managing Trace Volume with Jaeger Probabilistic Sampling

Reducing Storage Overhead with Probabilistic Sampling

Capturing every single request in a high-traffic microservices environment leads to massive storage costs and network congestion without providing additional diagnostic value. The solution is Probabilistic Sampling, which randomly selects a percentage of traces to record based on a defined probability. This ensures you have a statistically representative sample of system behavior without the overhead of 100% collection.

How Probabilistic Sampling Works

The Probabilistic sampler makes a decision based on the Trace ID. Because the decision is tied to the ID, every service involved in a specific request will make the same sampling decision. This prevents "broken traces" where one service records its span but a downstream service discards its own, which would otherwise leave gaps in your distributed trace.

Configuring the Sampler

In a standard Jaeger deployment, sampling is typically configured via environment variables passed to the jaeger-agent or the jaeger-collector. To implement a 1% sampling rate (meaning 1 out of every 100 traces is kept), use the following configuration.

Deployment Configuration:

# Run these as environment variables for the Jaeger component
export JAEGER_SAMPLER_TYPE=probabilistic
export JAEGER_SAMPLER_PARAM=0.01

Configuration Details:

• JAEGER_SAMPLER_TYPE: Set to probabilistic to enable random selection.
• JAEGER_SAMPLER_PARAM: A float between 0.0 and 1.0. 0.01 equals 1%, 0.1 equals 10%, and 1.0 collects everything.

Verification and Testing

To verify that your sampling rate is active, you can perform a volume test. Note that this requires a running Jaeger instance with the variables above applied.

1. Generate Traffic: Use a load-testing tool or a script to send a known number of requests (e.g., 10,000) through your instrumented application.
2. Query the UI: Open the Jaeger Query UI and search for traces within the time window of your test.
3. Analyze Results: If configured at 0.01, you should see approximately 100 traces.
4. Inspect Metadata: Click into a specific trace. In the trace details pane, verify that the Sampler field explicitly lists Probabilistic and the Sampler Param matches 0.01.

Limitations and Common Pitfalls

While effective, probabilistic sampling introduces specific risks that engineers must manage:

Risk	Impact	Mitigation
Rare Error Loss	Critical bugs occurring in 0.1% of requests may be sampled out and never recorded.	Use Adaptive Sampling or a higher rate during canary deployments.
Network Overhead	The agent may still forward spans to the collector before the collector drops them.	Ensure the sampler is configured as close to the application as possible.
Static Config	Changing JAEGER_SAMPLER_PARAM usually requires a restart of the component.	Use a configuration management tool (like Kubernetes ConfigMaps) to manage updates.

Rollback Procedure

If you find that critical data is missing and need to return to full tracing immediately:

1. Update the environment variable to JAEGER_SAMPLER_PARAM=1.0.
2. Restart the jaeger-agent and jaeger-collector pods or processes.
3. Verify in the UI that new traces are appearing at a 1:1 ratio with your requests.