OTLP Batch Exporter Latency Spikes Under Concurrent Load — Queue Saturation or Backoff?

Direct answers

1. Primary latency source: Most likely queue saturation, not scheduleDelay waiting or backoff. Under high concurrency, producer threads enqueue spans faster than the single export thread can flush them, so the bounded queue approaches maxQueueSize before maxExportBatchSize is reached. Once full, enqueue calls block or spans are dropped, and that wait dominates per-span export latency. Exporter backoff normally contributes only a few milliseconds per span when the collector is healthy.
2. Metrics to distinguish causes: Watch the exporter's queue-depth and drop signals (queue size near maxQueueSize, plus any dropped-spans counter) alongside export duration. If queue depth tracks the latency spikes, the cause is queue wait. If queue depth stays low while export duration spikes, suspect network round-trip time or retry/backoff. These are available from SDK instrumentation without debug logging, though exact metric names vary by SDK version and metrics backend.
3. Documented trade-off knobs: Yes. Reducing maxExportBatchSize flushes more often, lowering per-span latency at the cost of more export requests; increasing maxQueueSize reduces drop risk but lets more spans wait in the queue, which can raise latency under saturation. Pick based on whether drops or latency hurt you more.

Confirmed behavior (Java SDK 1.3x assumed)

• The BatchSpanProcessor runs a single worker thread that flushes when the batch reaches maxExportBatchSize or scheduleDelayMillis elapses, whichever comes first.
• The queue is bounded (default maxQueueSize is 2048 in the Java SDK). When full, new spans are dropped rather than queued indefinitely, and contention around the full queue shows up as latency.
• The OTLP exporter retries failed exports with exponential backoff. Backoff only matters when the collector returns errors or times out; it is not the usual driver of latency under normal operation.

Verification steps

1. Expose SDK metrics (for example a Prometheus /metrics endpoint) and generate the concurrent load that reproduces the spikes.
2. Correlate per-span export latency with queue depth and the dropped-spans counter. Latency rising as queue depth nears maxQueueSize confirms saturation.
3. Temporarily lower maxExportBatchSize (for example to 10). If latency falls while drops stay near zero, queue saturation was the primary cause.
4. To isolate backoff, make the OTLP endpoint unreachable or slow and check whether export duration rises while queue depth stays flat.

Assumptions and uncertainty

• Assumes a recent Java SDK (1.3x line). Other language SDKs differ in queueing, retry policy, and metric names; verify against your runtime.
• A collector with aggressive timeouts or its own overload can trigger frequent retries, making backoff the dominant factor instead.
• Custom or uninstrumented exporters may not emit the queue metrics described above.

One missing detail

What are your current maxQueueSize and maxExportBatchSize, and what queue depth do you observe at peak load? That determines whether shrinking the batch or growing the queue is the right first change.