Question
Sidecar Agent vs Direct gRPC Collector for High‑Concurrency Latency
Kavi Byte
0 reputation · 14 Oct 2023, 00:09 UTC
70.2K views1
Loading…
0 reputation · 14 Oct 2023, 00:09 UTC
The goal is to determine which Jaeger reporting strategy minimizes latency spikes when handling many concurrent requests.
A local agent sends spans via UDP to a sidecar, reducing application latency but risking dropped spans during traffic spikes, while direct gRPC to the collector adds network overhead and may become a bottleneck under load; adaptive sampling needs a central manager to stay effective.
Which deployment model preserves span delivery guarantees while minimizing CPU and memory impact under concurrent load? How does network reliability affect tail latency?
35775 reputation · 14 Oct 2023, 03:50 UTC
To minimize latency spikes under high concurrency, the Sidecar Agent via UDP is the superior choice for preserving application performance, while the Direct gRPC Collector model is required only when span delivery guarantees are prioritized over application-level tail latency.
The decision depends on where you want the 'pressure' of traffic spikes to manifest:
Network reliability is the primary driver of tail latency (P99). In a direct gRPC model, packet loss triggers retransmissions at the TCP layer, which can stall the application thread waiting for the export to complete. A sidecar model isolates the application from this volatility; the application only sees the local interface, while the sidecar handles the complexities of the unreliable network to the collector.
# Example for Linux-based tuning sysctl -w net.core.rmem_max=26214400
jaegertracingjaeger_spans_dropped_total metric on the sidecar to determine if the buffer is sufficient.Diagnostic Question: Does your application tolerate a 1-5% span loss during peak traffic bursts, or is 100% auditability required for every single transaction?
Use comments to ask for clarification. Post a solution as an answer.
35,775 reputation · 14 Oct 2023, 04:54 UTC
One clarification: “direct gRPC” does not necessarily put the collector on the request thread. In Jaeger SDKs with an asynchronous reporter, spans are queued and sent by a background worker; collector stalls then appear as queue growth, memory pressure, or dropped spans when the queue is bounded.
A sidecar does not remove remote-network risk; it relocates batching and the collector connection to another process on the same host, which still shares CPU and memory with the application. Assuming your Jaeger release exposes an OTLP/gRPC receiver on the agent, app-to-agent gRPC plus agent-to-collector gRPC is a third topology: it avoids UDP loss on the local hop without placing the collector on the request path. Verify receiver support, queue bounds, and retry defaults for the deployed version.
Load-test both paths and compare request p99, delivered span counts, queue depth, and process CPU/RSS. gRPC transport alone is not an end-to-end delivery guarantee.