OTLP Batch Exporter Latency Spikes Under Concurrent Load — Queue Saturation or Backoff?
0 reputation · 18 Sept 2022, 02:53 UTC
Observed behavior
When an application using the OpenTelemetry SDK with the OTLP exporter handles many concurrent requests, per-span export latency rises sharply compared to sequential traffic. The BatchSpanProcessor is documented to buffer spans and flush them when either maxExportBatchSize is reached or scheduleDelayMillis elapses, but under concurrency it is unclear which limit is being hit first and how the exporter's queue-full condition interacts with request threads.
Constraints and uncertainty
The documented failure condition SpanExporter dropping spans when the queue is full is well defined, but the latency contribution of exporter retry/backoff versus queue wait time is not clearly separated in the SDK metrics. Assume a recent stable SDK release (Java 1.3x or equivalent); behavior may differ across language SDKs and collector versions, so results need verification against the specific runtime in use.
Questions
- Under concurrent load, does BatchSpanProcessor latency come primarily from
scheduleDelaywaiting, queue contention, or OTLP export retries with backoff? - Which SDK or collector metrics can distinguish queue saturation from network round-trip time without enabling debug logging?
- Is reducing batch size or increasing
maxQueueSizethe documented way to trade latency for drop risk under concurrency?