Controlling Trace Volume with Jaeger Remote Sampling

The High-Traffic Trace Dilemma

In a production microservices environment, tracing every single request is rarely sustainable. High-traffic endpoints can generate gigabytes of telemetry per minute, leading to storage exhaustion and significant network overhead. However, hardcoding a sampling rate (e.g., 1%) into your service configuration creates a rigid system: you cannot increase visibility during an incident without a full redeploy, and you cannot lower the rate for a noisy endpoint without affecting the entire service.

The solution is Remote Sampling. Instead of the service deciding its own sampling rate, it polls a central Jaeger collector to receive sampling strategies. This allows you to adjust trace volume across your entire fleet in real-time from a single configuration file.

How Head-Based Remote Sampling Works

Jaeger primarily uses head-based sampling. This means the decision to sample a trace is made at the very beginning of the request (the root span). This decision is then propagated through the trace context to every downstream service. If the root service decides not to sample, no spans for that request are generated or sent, saving resources across the entire call chain.

With remote sampling, the Jaeger collector acts as a strategy server. The client SDKs periodically call a specific endpoint on the collector, identifying themselves by service name. The collector responds with a JSON strategy that tells the SDK exactly how to handle requests.

The Strategy Hierarchy

Remote sampling strategies typically follow a two-tier logic:

• Per-Operation Rules: Specific rules for high-volume or critical endpoints (e.g., "sample /healthz at 0.001% but sample /checkout at 10%").
• Default Strategy: A fallback probability used for any operation not explicitly listed in the per-operation rules.

Practical Implementation: The Strategies File

To implement this, you define a sampling strategy file (usually strategies) and point your Jaeger collector to it. Below is an example of a strategy configuration designed for a typical e-commerce backend.

{
  "default": {
    "type": "probabilistic",
    "param": [0.001]
  },
  "services": {
    "order-service": {
      "operations": {
        "createOrder": {
          "type": "probabilistic",
          "param": [0.1]
        },
        "getOrderHistory": {
          "type": "probabilistic",
          "param": [0.01]
        }
      }
    }
  }
}

Configuration Breakdown:

• The default rate is set to 0.1% to keep baseline noise low.
• The createOrder operation is sampled at 10%, ensuring high visibility into the most critical business transaction.
• The getOrderHistory operation is sampled at 1%, providing a balance between visibility and volume.

Verification and Diagnostics

Because remote sampling relies on a polling mechanism, changes are not instantaneous. There is a propagation delay based on the SDK's poll interval.

Verifying the Strategy

You can verify that the collector is serving the correct strategy without waiting for the SDK to poll. Run this command from a terminal with access to the collector (replace jaeger-collector with your actual hostname):

# Run as a standard user with curl permissions
curl http://jaeger-collector:14268/strategies/order-service

Expected Result: The collector should return the JSON fragment corresponding to the order-service configuration. If it returns a 404 or a default empty strategy, check the collector's startup logs for errors related to loading the strategies file.

Validating Trace Volume

To confirm the strategy is working in practice, generate a burst of traffic to a specific endpoint and check the Jaeger UI. If you configured a 10% rate for createOrder, roughly 10 out of 100 requests should appear as traces. If 0% or 100% appear, the SDK is likely falling back to a local default because it cannot reach the collector.

Trade-offs and Limitations

Remote sampling is powerful, but it has a critical blind spot: it cannot react to errors.

Because the decision is made at the "head" of the trace, if a request is not sampled and subsequently fails in a downstream service, that failure is lost to the tracing system. You cannot "retroactively" decide to keep a trace because it ended in an error. To mitigate this, you should pair remote sampling with:

• Structured Logging: Ensure every error is logged with its trace_id. Even if the trace isn't in Jaeger, the log provides the evidence.
• Metrics: Use Prometheus or similar tools to track error rates, which should trigger the human decision to increase the sampling rate via the remote strategy file.

Closing Action

To move toward centralized trace control, start by auditing your highest-volume endpoints. Define a conservative default sampling rate and create explicit per-operation rules for your most critical paths. Once the strategy file is deployed to your collector, verify the endpoint via curl before monitoring the trace volume in your UI.