Jaeger Trace Sampling: Choosing Probabilistic, Rate Limiting, or Hybrid

Every request in a distributed system can produce spans—timed operations within a service—across several services. A trace is the sequence of spans that follows one request through the system. Without sampling, trace storage grows in direct proportion to request volume and can quickly exceed the capacity of Elasticsearch, OpenSearch, or Cassandra backends. The practical problem is not whether to collect traces, but how to bound ingestion so that storage stays affordable while rare failure paths remain visible.

Useful takeaway: start with probabilistic sampling as the baseline for steady coverage, add a rate-limiting ceiling for high-traffic or bursty services, and verify the active strategy through the collector’s /sampling endpoint before relying on it in production.

Decision and constraints

The decision is which sampling strategy to assign to each service, and what parameter to use. Treat it as a capacity decision, not just an observability preference.

• Storage budget: maximum spans per day retained by the backend.
• Burst tolerance: whether the collector can absorb traffic spikes without dropping all traces.
• Low-volume visibility: whether infrequent services still need sampled traces for incident investigation.
• SDK support: whether the client libraries in use can fetch and apply a remote sampling strategy.

Supported sampling options

Strategy	How it works	Best suited for	Main risk
Probabilistic	Selects a fixed percentage of traces based on the trace ID, giving stable coverage across services.	General traffic and cross-service correlation where predictable storage growth matters.	A rate that is too low can miss rare failure paths.
Rate limiting	Caps the number of traces per second per service, protecting the collector during bursts.	High-cardinality (many distinct services or identifiers) or bursty services that could otherwise dominate ingestion.	A noisy high-traffic service can consume the quota and starve low-volume services.
Hybrid	Applies probabilistic sampling as a baseline and rate limiting as an upper bound.	Most production systems that need both steady coverage and burst protection.	More parameters to tune per service.

Trade-offs

Probabilistic sampling spreads coverage evenly because the decision is derived from the trace ID. The same trace is sampled consistently across services, which preserves end-to-end trace completeness. The downside is statistical: rare errors that occur in a small fraction of traces may never be sampled if the percentage is too aggressive.

Rate limiting protects the collector from CPU and network saturation during traffic spikes because it stops accepting new trace decisions after the per-second cap is reached. The downside is locality: the cap applies per service, so one chatty service can use the entire budget while a business-critical but low-traffic service receives no samples.

A hybrid configuration attempts to combine both behaviors: probabilistic sampling handles normal load, while rate limiting acts as a ceiling. This is usually the most practical production choice, but it requires two parameters per service and more careful validation.

Example configuration and validation

Version assumptions: this guide targets the Jaeger 1.x collector using a JSON strategies file. OpenTelemetry Collector and Jaeger OpenTelemetry SDKs use different sampler semantics; do not port these values directly during migration.

Create a strategies file such as <STRATEGIES_FILE_PATH>. This is an example structure; confirm the exact JSON schema against your Jaeger release.

{
  "default_strategy": {
    "type": "probabilistic",
    "param": 0.01
  },
  "service_strategies": {
    "payments-service": {
      "type": "ratelimiting",
      "param": 5
    }
  }
}

Start the collector with the strategies file, for example:

jaeger-collector \
  --sampling.strategies-file=<STRATEGIES_FILE_PATH> \
  --es.server-urls=http://elasticsearch:9200

Where to run: on the collector host or in the collector container. Required permissions: ability to edit the collector configuration or ConfigMap and restart the collector process or pods. Risk: restarting the collector creates a brief gap in span ingestion.

Verify the strategy by querying the collector’s HTTP sampling endpoint:

curl -s "<COLLECTOR_HTTP_BASE>/sampling?service=payments-service"

Expected check: the response should indicate the intended strategy type and parameter, for example a probabilistic rate of 0.01 or a rate limit of 5 traces per second. Exact JSON field names vary by Jaeger version, so inspect the actual response rather than assuming a schema.

To validate behavior under load, generate traffic above the configured rate and compare emitted spans against stored traces in the Jaeger UI. If the rate limit is 5 traces per second, a one-minute burst should produce roughly no more than 300 stored traces for that service, assuming clients honor the remote strategy.

Inspect client SDK logs for successful sampling strategy retrieval and the sampler type applied to new spans. Sampling strategy updates are not instantaneous across all client SDKs; confirm that the specific library supports dynamic re-fetching before treating the change as live.

Limitations and practical verification

• Rate limiting is per service, not global, so a single service can dominate its own quota.
• Too-low probabilistic rates reduce the value of tracing for incident investigation.
• Client SDKs cache the sampling decision, so strategy changes take time to propagate.
• Jaeger sampling semantics differ from OpenTelemetry sampler configuration.

Practical way to check the result: open the Jaeger UI, query the service over a known time window, and compare the trace count against the expected rate multiplied by the duration. If the count is far above or below the expectation, revisit the strategies file and client SDK configuration.

Rollback

Because changing the collector strategies file changes system state, rollback by restoring the previous strategies file and restarting the collector. Confirm the /sampling endpoint returns the previous strategy before resuming normal traffic shaping.