Controlling Jaeger trace storage cost with remote sampling

The problem: trace storage grows faster than your budget

A single high-traffic endpoint can generate millions of spans per day. If your Jaeger client exports every span, storage costs and query latency rise in direct proportion to traffic. You still need traces for latency spikes and rare errors, but storing everything is rarely affordable.

The useful takeaway is to let the Jaeger agent make the sampling decision. Configure a remote sampling strategy, verify it by querying the agent endpoint, and treat unsampled traces as invisible rather than missing data.

Choose a sampling strategy

A span is a timed unit of work; a trace is a chain of spans across services. Sampling decides whether a trace is exported and stored.

• Probabilistic sampling keeps a fixed percentage of traces. Cost is predictable, but low-traffic services may never sample a rare error or slow path.
• Rate-limiting sampling caps new traces per second for a service. It protects hot entry points from trace storms, but the limit is typically applied per service instance, so many instances can still exceed the intended aggregate volume.
• Tail-based sampling keeps traces based on errors, latency, or attributes after a trace finishes. It is useful for debugging, but it buffers in-progress traces and is not suitable for alerting on incomplete requests.

Classic Jaeger agents serve a sampling strategy from a JSON file, while Jaeger distributions built on the OpenTelemetry Collector may use collector processors instead. Check your distribution and version before changing configuration.

Worked example: verify remote sampling on the agent

This example assumes a Jaeger 1.x agent with the classic sampling endpoint and a client library that supports the remote sampler.

1. Define the sampling strategy

Create a strategies file, for example /etc/jaeger/sampling-strategies.json:

{
  "serviceStrategies": [
    {
      "service": "orders-api",
      "type": "probabilistic",
      "param": 0.1
    },
    {
      "service": "checkout-api",
      "type": "ratelimiting",
      "param": 5
    }
  ]
}

This keeps roughly 10 percent of orders-api traces and caps checkout-api at five new traces per second per agent instance.

2. Start the agent with the strategy file

Run the agent with the flag that loads the file. Flag names vary by Jaeger version; consult your release notes.

jaeger-agent --sampling.strategies-file=/etc/jaeger/sampling-strategies.json

3. Query the agent endpoint

From any machine that can reach the agent, run:

curl -s http://JAEGER_AGENT_HOST:5778/sampling?service=SERVICE_NAME

Required permissions: none beyond network access to the agent port. The expected response is a JSON object describing the strategy, such as the type and rate. Risks: a connection refused error means the wrong host, port, or agent version; an empty or unexpected strategy means clients may fall back to a local default.

4. Configure the client

Set the client to use remote sampling. Environment variable names differ by language library, but the pattern is typically:

JAEGER_SAMPLER_TYPE=remote
JAEGER_SAMPLER_MANAGER_HOST_PORT=JAEGER_AGENT_HOST:5778

5. Verify with a controlled burst

Send a known number of requests with a unique tag, then count matching traces in Jaeger query. At a 10 percent probabilistic rate, 1,000 requests should produce roughly 100 traces, with statistical variance. If the count is near zero, inspect client logs for sampling strategy responses; the client may not have fetched the agent strategy. If the count is far above a rate limit, you may be sending traffic to multiple agent instances, each applying its own limit.

Trade-offs and limitations

• Unsampled traces are invisible to Jaeger queries and service graphs. Do not use sampled trace counts as exact denominators for SLOs or error budgets.
• Rate limits are typically per agent instance or per service instance, not cluster-wide. Aggregate trace volume can exceed a single configured limit.
• Do not hardcode a sampler in application code while also allowing the agent to override it. Configuration drift can silently change trace volume.
• Tail-based sampling delays the decision until trace completion, increasing memory usage, and cannot alert on in-flight requests.

Closing checklist

1. Confirm your Jaeger distribution and version: classic agent sampling and OpenTelemetry Collector-based sampling are configured differently.
2. Deploy the sampling-strategies file and restart the agent.
3. Query /sampling?service= for every instrumented service and confirm the returned strategy matches your intent.
4. Run a controlled burst and compare observed trace volume to the configured rate or limit.
5. Document that dashboards and alerts based on trace counts exclude unsampled traffic.

If you need to revert, restore the previous strategies file and restart the agent; clients will pick up the old strategy on the next poll.