Architecting Adaptive Sampling in Jaeger to Prevent Backend Overload

In high-traffic microservice environments, capturing every single trace is often a recipe for database failure or prohibitive storage costs. Static sampling rates are frequently too blunt: they either miss critical errors during traffic spikes or overwhelm the system during bursts. Adaptive sampling solves this by allowing the Jaeger collector to dynamically adjust sampling rates based on real-time throughput, ensuring a statistically significant sample of system behavior without manual intervention.

Requirements for Adaptive Sampling

To implement adaptive sampling effectively, the architecture must satisfy three core requirements:

• Centralized Sampler: A component that calculates the optimal sampling rate based on the target throughput versus current incoming load.
• Client-Side Awareness: Libraries within services that can poll the sampler for updated strategies and apply them locally.
• Context Propagation: A mechanism, typically W3C Trace Context or B3 headers, to ensure that if a trace is sampled at the entry point, all downstream services respect that decision to maintain trace integrity.

The Smallest Suitable Design

The most viable architecture for most teams involves the Jaeger Remote Sampler. In this design, application instances do not make sampling decisions in isolation; instead, they fetch a sampling strategy from the Jaeger collector at regular intervals.

The workflow follows this sequence:

1. The operator defines a target traces per second (TPS) (e.g., 100 traces/sec).
2. The Jaeger collector monitors the actual request rate across all instances.
3. The sampler service calculates the required probability percentage to hit that target.
4. Application instances poll the sampler (e.g., every 1-10 seconds) and update their internal sampling logic.

Data Boundaries and Integrity

The primary data boundary exists between the application runtime and the Jaeger infrastructure. The application is responsible for injecting sampling headers into requests, while the infrastructure is responsible for the policy.

A critical risk occurs when high-cardinality tags—such as unique user IDs or high-precision timestamps—are used in spans. If the polling interval is too short and traffic is extreme, the collector may experience memory exhaustion trying to track state for thousands of permutations. Ensure that sampling logic is decoupled from high-cardinality business metadata.

Operational Checks

To ensure the adaptive logic is functioning correctly, monitor these specific metrics provided by the Jaeger collector and client libraries:

Metric Name	Purpose
remote_sampler_rate	The current sampling probability being applied by clients.
collector_span_latency	Indicates if the backend is struggling to process the sampled volume.
sampler_poll_errors	Tracks how often clients fail to reach the sampler service.

Failure Modes and Fallback Strategies

The system must be resilient to the failure of the sampler service. If the sampler becomes unreachable, client-side libraries must not stop tracing entirely or crash the application.

• Static Fallback: Always configure a fallback rate (e.g., 0.01 for 1%) in environment variables. This is used if the remote sampler times out.
• Oscillation: If the target throughput is set too low, the sampling rate may fluctuate wildly between 0% and 1%, leading to inconsistent trace data. Increase the polling interval to smooth out these transitions.

Verification Procedure

To verify the configuration, perform a simulated load test using a tool like k6 or Locust. Start with low-volume traffic and set a target of 50 traces per second. Monitor the remote_sampler_rate metric in your dashboard; the rate should drop as load increases to maintain the 50 TPS target.

To test failure resilience, simulate a network partition between the application and the sampler. Verify that the client-side libraries transition to the defined static fallback rate rather than dropping all traces or failing requests.

Rollback Procedure

If adaptive sampling causes instability or memory exhaustion in the collector, revert the client configuration to a constant sampling strategy. This removes the dependency on the remote sampler and restores a predictable, fixed load on the backend.