Batch Span Export Version 1.25.x Limits Alert Noise in OpenTelemetry

Direct answer

Set sampler probability per resource using observed span-rate percentiles rather than a fixed rate, run a separate BatchSpanProcessor (or resource-scoped exporter schedule) for each high-variance resource, and switch alert thresholds from absolute span counts to dynamic percentiles such as the 95th percentile per window. This reduces batch-flush noise without suppressing real anomalies.

Likely explanation versus established behavior

Likely explanation: Intermittent alert suppression happens when a static sampler probability meets uneven traffic. During a spike, a low probability drops too many spans; during a lull, the same probability exports enough spans to look like normal traffic, so threshold crossings are missed or delayed. Batch flush timing can line up with these drops, making suppression look intermittent.

Established behavior for 1.25.x:

• Parent-based and probabilistic sampler precedence interacts with BatchSpanProcessor flush timing; misaligned settings can suppress alerts when traffic varies across resources.
• A single BatchSpanProcessor applies one flush interval to every resource sharing that exporter instance.
• The exporter does not support per-resource retry policies; retry and timeout values are shared across all resources using the same exporter.
• Dynamic percentile thresholds absorb traffic variance better than absolute counts.

Steps for this case

1. Measure span creation rates per resource over a representative window and note the 50th and 95th percentiles.
2. Configure a probabilistic sampler per resource with probability scaled to the resource's variance: higher probability for high-variance or critical resources, lower for steady, low-signal resources.
3. Isolate batch export by creating a dedicated BatchSpanProcessor per resource, or by assigning resource-scoped schedule and timeout settings where the SDK supports them. Do not assume per-resource retries are possible.
4. Replace absolute alert thresholds with a rolling 95th-percentile span-count threshold per resource, alerting only when the current window exceeds the baseline percentile by a defined margin.
5. Validate under controlled traffic: hold sampler probability constant, enable OTLP debug logging, and confirm alert firings correlate with percentile thresholds rather than batch flush events.

Missing diagnostic detail

Which sampler type is currently configured—probabilistic, parent-based, or deterministic—and what are the current exporter retry and timeout values? This changes whether per-resource isolation is safe and how aggressively thresholds can be tightened.

Version assumption

Behavior below assumes OpenTelemetry SDK 1.25.x. Flush interval and sampler interaction can differ between minor patches, so validate against the exact build in production before rolling out threshold changes.