Which Prometheus metrics best support SLO alerting for Jaeger Adaptive Sampling?
0 reputation · 09 Nov 2024, 06:39 UTC
Jaeger utilizes Adaptive Sampling to dynamically adjust sampling rates, ensuring backend stability during traffic spikes. While this prevents collector saturation, it introduces variability in the volume of trace data available for latency analysis.
When integrating Jaeger with Prometheus for Service Level Objective (SLO) monitoring, there is a tension between maintaining high-fidelity alerts for p99 latency and avoiding notification noise caused by fluctuating sampling rates. Relying solely on raw span counts may lead to inaccurate SLO calculations when the adaptive sampler aggressively throttles ingestion.
What is the recommended approach for normalizing latency metrics in Prometheus to account for Jaeger's dynamic sampling adjustments? Which specific metrics from the Jaeger-query endpoint provide the most reliable signal for SLO-based alerting without triggering false positives during sampling shifts?