Jaeger Collector gRPC Connection Pool: Unclear Backpressure Policy on Stream Exhaustion

Stream Exhaustion Behavior

When the Jaeger Collector reaches the limit defined by grpc-server.max-concurrent-streams, the underlying gRPC-Go server returns a RESOURCE_EXHAUSTED error (corresponding to an HTTP/2 REFUSED_STREAM) to the client. The collector does not silently discard spans at this stage; the rejection occurs at the transport layer before the spans are deserialized or processed by the application.

Distinguishing Stream Rejection from Span Dropping

It is critical to distinguish between stream exhaustion and internal queue overflow, as they manifest differently in metrics:

• Stream Exhaustion: Occurs at the gRPC layer. The client receives RESOURCE_EXHAUSTED. This is not tracked by jaeger_collector_spans_dropped_total because the spans never entered the collector's processing pipeline.
• Internal Queue Overflow: Occurs after the stream is accepted. If the internal processor queue (collector.queue-size) is full, spans are dropped and recorded in the jaeger_collector_spans_dropped_total metric. This is often misinterpreted as sampling but is actually a backpressure failure.

Tuning and Resource Bounds

Because each active stream requires a handler goroutine and a read buffer (approximately 4 KB), setting the limit too high can lead to memory exhaustion (OOM) or CPU contention during bursts.

Recommended Guidelines:

• Baseline: For a standard 4-core collector instance, a limit between 500 and 1,000 is typically sufficient for high-throughput ingestion.
• Upper Bound: Avoid exceeding 2,000 concurrent streams unless the instance is heavily scaled in terms of CPU and RAM, as goroutine scheduling overhead begins to impact latency.
• Client Strategy: To prevent the "reconnection storms" mentioned, ensure clients implement exponential backoff with jitter when encountering RESOURCE_EXHAUSTED.

Verification Steps

To confirm if stream exhaustion is occurring in your environment, use the following methods:

1. Prometheus Query: Check for gRPC-level errors (requires gRPC-Go telemetry enabled):

   increase(grpc_server_handled_total{grpc_code="RESOURCE_EXHAUSTED"}[5m])

2. Channelz Inspection: If enabled, inspect the /debug/channelz endpoint to monitor the real-time count of active streams against your configured limit.
3. Log Analysis: Set the collector log level to debug and monitor for REFUSED_STREAM or gRPC server limit warnings.

Missing Diagnostic Detail: Are you running the collector behind a load balancer or service mesh (e.g., Envoy, Istio)? If so, the proxy may be enforcing its own max_concurrent_streams limit, which would trigger these errors before the request ever reaches the Jaeger Collector.