Connection pool limits causing intermittent span export failures in Jaeger

Answer to the Core Questions

1. What is the optimal maximum connection count for sustained high throughput?

There is no single magic number. The right pool size is the smallest number that keeps the exporter’s p99 export latency flat while providing headroom for retries and collector‑side back‑pressure. A practical sizing rule is Little’s Law for the exporter:

PoolSize ≈ (SpansPerSecond × AvgExportLatency) + Headroom

Where Headroom is usually 20–30 % of the computed product to absorb bursty traffic and retry spikes. Verify this with a controlled load test instead of adopting the default 50 connections.

2. How does the exporter decide to drop spans versus retrying when the queue is full?

• Spans are first pushed into a bounded in‑memory queue. If the queue is already full, the exporter drops the span immediately and increments a spans_dropped counter.
• If the span is enqueued successfully, the exporter attempts to send it to the collector. On failure it retries up to maxRetries times, waiting retryInterval between attempts.
• After the retry limit is reached, the span is dropped and the spans_dropped counter is incremented again.

Note that the exact names of the counters and the order of operations vary between language SDKs and Jaeger releases, so always consult the documentation for the specific version you are using.

3. Is there a documented back‑pressure mechanism to prevent pool exhaustion when the collector is overloaded?

Jaeger’s legacy clients expose no true application‑level back‑pressure. The exporter’s queue is a simple drop‑on‑full buffer, and retries can amplify the problem. The recommended mitigations are:

• Use a higher‑level OpenTelemetry pipeline, which offers a BatchSpanProcessor with configurable queue size, batch size, and export timeout.
• Apply adaptive sampling or a static sampling rate to reduce the number of spans that reach the exporter.
• Move the export logic into a collector‑side component (e.g., a separate ingestion queue) that can provide back‑pressure signals.
• Monitor collector metrics (accepted, refused, dropped) and scale the collector or its resources before adjusting the client pool.

Step‑by‑Step Tuning Procedure

1. Identify the transport. Check your client configuration to confirm whether you are using HTTP, gRPC, or UDP. The "connection pool exhausted" warning only appears with HTTP or gRPC.
2. Check SDK version and defaults. Open the SDK’s documentation for your exact release to read the default pool size, queue capacity, and retry limits.
3. Enable metrics. Turn on the exporter’s internal metrics (e.g., exporter_spans_exported, exporter_spans_dropped, exporter_queue_size, exporter_connections_in_use) and observe them under a steady load.
4. Run a stepped load test. Increase the span rate incrementally, record the p99 export latency, the number of connections in use, and the dropped‑span counter. Stop when latency starts to rise.
5. Compute pool size. Use the formula above with the measured SpansPerSecond and AvgExportLatency.
6. Apply the new pool size. Reconfigure the exporter (e.g., maxConnections=200 for the HTTP client) and repeat the load test to confirm stability.
7. Adjust retry policy. If retries are causing repeated pool occupation, reduce maxRetries or increase retryInterval with jitter.
8. Consider back‑pressure alternatives. If the collector remains a bottleneck, enable adaptive sampling or migrate to an OpenTelemetry pipeline with a bounded queue.

Missing Diagnostic Detail Needed

To give a precise recommendation for your environment, could you tell me which Jaeger client SDK (e.g., Java, Go, Python) and version you are using, and whether you are sending via HTTP, gRPC, or UDP?