Centralize Jaeger Trace Sampling Without Redeploying Services

Sampling should not require a redeploy

You instrumented your services with Jaeger, but each tracer starts with a hard-coded sampling probability. When traffic spikes or an incident needs deeper visibility, changing that probability usually means editing code, rebuilding images, and redeploying every affected service.

The useful takeaway is that Jaeger’s collector can serve a sampling strategy document that client tracers poll. Operators can tighten or loosen trace volume from one central place without touching application deployments.

How Jaeger remote sampling works

A trace is the full path of a request through distributed services. A span is one timed operation inside that trace. The sampler decides whether to start a trace and whether to keep adding spans to it. The collector receives spans from clients and stores them for the query service.

With remote sampling, the client asks the collector at an endpoint such as /api/sampling for a strategy document. The document can include:

• Probabilistic sampling: keep a fixed percentage of traces.
• Rate limiting: keep a maximum number of traces per second.
• Per-operation overrides: different rates for specific span names, with a default fallback.

The client applies the strategy locally. The collector and query service only see spans the client chooses to send.

Worked example: central sampling for orders-api

Suppose you have a service named orders-api and a Jaeger collector reachable at . You want to keep 10 percent of traces by default, limit overall volume to five traces per second, and sample every checkout operation.

Create a strategy file at on the collector host:

{
  "strategyType": "probabilistic",
  "probabilisticSampling": {
    "samplingRate": 0.1
  },
  "rateLimitingSampling": {
    "maxTracesPerSecond": 5
  },
  "operations": [
    {
      "operation": "checkout",
      "probabilisticSampling": {
        "samplingRate": 1.0
      }
    }
  ],
  "defaultSampling": {
    "probabilisticSampling": {
      "samplingRate": 0.1
    }
  }
}

Point the collector to that file in its configuration, then restart the collector. Run this on the host where the collector process runs, with permissions to edit its config directory and restart the service:

sudo systemctl restart jaeger-collector

Configure the orders-api tracer to use remote sampling. Exact environment variables vary by Jaeger client library, but a typical setup uses a remote sampler type and an endpoint such as:

JAEGER_SAMPLER_TYPE=remote
JAEGER_SAMPLING_ENDPOINT=http://:5778/api/sampling?service=orders-api

Verify the collector is serving the strategy. Run this from any machine with network access to the collector:

curl -s http://:5778/api/sampling?service=orders-api

Expected check: an HTTP 200 response containing JSON that matches the strategy file. A risk is that restarting the collector briefly stops serving strategies; clients that poll during the window may fall back to a local default.

Trade-offs and limitations

• SDK support for remote sampling fields varies by language. Test your exact client library before relying on per-operation overrides.
• When the collector is unreachable, clients may fall back to a local default sampler, which can produce unexpectedly high or low trace volume.
• Lowering the sampling rate reduces future spans only. Already discarded spans cannot be recovered.
• Storage backend and retention settings determine how long sampled traces remain queryable. Size storage for the expected sampled volume, not the raw request volume.

Verify before you rely on it

1. Run Jaeger locally and attach a test instrumented service.
2. Record the trace count for orders-api in the Jaeger UI over a fixed time window.
3. Change the remote sampling rate in the strategy file and restart the collector.
4. Wait for clients to poll the new strategy, then compare trace counts for the same service and window. The count should move in the direction of the new rate.
5. Stop the collector or block client access to port 5778. Check SDK logs for a fallback sampler message and observe the resulting trace volume.

A practical check is to compare the UI trace count before and after the change, and to confirm that fallback behavior is acceptable during collector outages.

Closing

Remote sampling separates trace volume decisions from application releases. Start with a probabilistic default, verify that your client SDK honors the strategy, document the fallback behavior, and size storage for the sampled volume you actually expect to keep.