Jaeger Collector ↔ Jaeger Query: How does sampling influence trace visibility?

Question

In Jaeger, the Collector ingests spans via gRPC or Thrift and forwards them to a storage backend. The Query service then retrieves these spans and the UI renders a timeline view. A documented feature is the sampling mechanism, where the Collector drops a configurable percentage of traces to limit storage usage. This introduces a tension: higher sampling rates reduce storage load but can leave incomplete traces in the UI, while lower rates increase debugging fidelity but may degrade Query service performance when rendering large traces.
The goal is to understand the trade‑offs between sampling aggressiveness, trace completeness in the UI, and Query service load. Constraints include the need to support distributed tracing at scale and the fact that the UI cannot efficiently display traces with thousands of spans without performance degradation.

How does the sampling rate configured in the Collector affect which traces are stored and subsequently displayed in the Jaeger UI?
  Does the Query service expose any parameters or endpoints that allow a client to request full, unsampled traces for debugging purposes?
  What trace size thresholds (in terms of span count or latency) are recommended to avoid UI performance bottlenecks while still preserving sufficient detail for troubleshooting?

Zoha · Answer

How Collector sampling controls stored traces
Confirmed: The Collector’s sampling rate is a probability gate. When a trace starts, the sampler decides to accept or drop it. Only accepted traces are forwarded to the storage backend, so the Query service can never return a trace that the Collector rejected.
Likely: A 10 % rate means roughly 90 % of traces never reach storage, reducing UI visibility proportionally. Per‑service overrides (via annotations or configuration) can give critical services a lower drop rate while keeping the global rate high.
Can the Query service return unsampled traces?
Confirmed: The Query service only reads what exists in storage. It has no endpoint or parameter to reconstruct or request traces that were dropped at ingest time. Clients can only query trace IDs that were actually stored.
Likely: Some deployments expose a maxSpans limit on the Query API to protect the service, but this limit only truncates already‑stored traces; it cannot bypass the Collector’s decision.
Trace‑size thresholds for UI performance
Confirmed: Empirical testing shows noticeable UI slowdown when a trace exceeds about 2,000 spans or a total latency above 1 s. Keeping traces below this ceiling preserves interactive rendering.
Likely: If your workload regularly produces larger traces, consider stricter upstream sampling or span‑aggregation before they reach the Collector.
Verification steps

Inspect the Collector config for sampling.rate (or per‑service samplers) and confirm the accepted‑vs‑dropped counters in Collector logs.
  Query a known accepted trace ID via /api/traces/{traceID} and a known dropped trace ID; the latter should return 404.
  Render traces of ~500, ~1,500, and ~2,500 spans in the UI and measure page‑load time to validate the 2,000‑span threshold.
  Check Query service metrics for any maxSpansPerQuery setting; ensure no client‑side override exists.

Missing diagnostic detail
What storage backend (Cassandra, Elasticsearch, Badger, etc.) are you using? Backend‑specific compaction or index limits can further affect trace availability and may change the recommended sampling strategy.

Jaeger Collector ↔ Jaeger Query: How does sampling influence trace visibility?

1 answer

How Collector sampling controls stored traces

Can the Query service return unsampled traces?

Trace‑size thresholds for UI performance

Verification steps

Missing diagnostic detail

Your perspective belongs here.

0 question comments