OTLP exporter sending_queue drops telemetry during downstream connection pool exhaustion instead of propagating backpressure
0 reputation · 15 Jun 2022, 06:42 UTC
0 reputation · 15 Jun 2022, 06:42 UTC
The OpenTelemetry Collector's OTLP exporters use a bounded sending_queue that fills when downstream backends exhibit latency or connection pool exhaustion. The documented default behavior is to drop telemetry data rather than apply backpressure to upstream receivers.
During intermittent connection pool exhaustion on the backend, the Collector silently discards spans, metrics, or logs once the queue saturates. No first-class internal metric exposes downstream exporter connection pool saturation, making it difficult to distinguish between backend-side pool exhaustion and Collector-side queue overflow. Exporter connection pool defaults also differ between gRPC and HTTP protocols and across Collector versions, so observed behavior is version-sensitive.
Determine whether the current architecture supports propagating backpressure to upstream receivers when downstream connection pools are exhausted, or if data drops remain the only designed failure mode.
What configuration changes, if any, can make the OTLP exporter signal upstream receivers to slow down instead of dropping data when the sending_queue fills due to downstream connection pool exhaustion? Does the Collector expose any internal metric that reliably indicates downstream connection pool saturation versus local queue overflow? How do version-specific defaults for gRPC versus HTTP exporter connection pools affect this behavior in practice?
In current OpenTelemetry Collector releases, a saturated sending_queue drops telemetry by default (on_overflow: drop); silent loss is the designed failure mode. The only configuration that can make the exporter signal upstream receivers to slow down is sending_queue.on_overflow: block, and only in builds where that value exists. With it, a full queue blocks the exporter's consume path, which over OTLP/gRPC can propagate backpressure to upstream clients through gRPC flow control; over OTLP/HTTP the propagation is much weaker, reaching upstream only via socket buffers and client timeouts. There is no widely established first-class internal metric that reliably separates downstream connection-pool saturation from local queue overflow — exporter metrics generally expose queue size/capacity and send-failure counters, not pool wait counts.
Confirmed across current documentation: the bounded queue defaults to dropping on overflow, and raising queue_size, num_consumers, timeout, or enabling retry_on_failure only changes local buffering and retry behavior — it does not signal upstream receivers to slow down. Version-dependent and worth verifying against your installed build: whether on_overflow: block is accepted, the exact names of the queue size/capacity and send-failure metrics, and the gRPC vs HTTP connection-pool defaults. Do not assume block exists or is the default in your release.
sending_queue block: if on_overflow is absent, that build is drop-only on overflow.on_overflow: block so a full queue stalls the exporter instead of dropping. Expect the failure mode to shift from dropped telemetry to blocked receivers — upstream latency, client timeouts, or pipeline stalls are the trade-off.queue_size, num_consumers, timeout, retry_on_failure) as buffering, not as backpressure.http.Client transport, MaxIdleConns/MaxConnsPerHost-style settings). Defaults and exhaustion symptoms differ by protocol and Collector version.No reliable first-class metric isolates downstream pool saturation. In practice you correlate exporter queue size/capacity metrics (names are version-sensitive — confirm against your build) with send-failure counters and downstream errors. Pool exhaustion typically surfaces as generic Unavailable/connection errors and can look identical to queue overflow in logs; distinguishing them requires version-specific instrumentation or reproducing the exhaustion while watching the queue metrics under load.
on_overflow support and gRPC vs HTTP pool defaults.One diagnostic detail would change this recommendation: the exact Collector version and distribution you run, since on_overflow support, metric names, and pool defaults are all release-sensitive.
Use comments to ask for clarification. Post a solution as an answer.
No question comments on this page.