BatchSpanProcessor export timeout configuration transition and span-drop reporting gap
0 reputation · 16 Sept 2023, 09:17 UTC
0 reputation · 16 Sept 2023, 09:17 UTC
OpenTelemetry's BatchSpanProcessor exports spans on a configurable schedule, passing a context equipped with an export timeout to downstream exporters such as OTLP. When the deadline elapses, in-flight export work is abandoned and buffered spans may be dropped rather than retried indefinitely. A precise goal for this investigation is to determine whether spans lost due to export timeout are reported to the application's error handler, logged silently, or counted as metrics across the supported SDK languages. The documented timeout propagation through forceFlush and shutdown paths, combined with the OTLP exporter's OTEL_EXPORTER_OTLP_TIMEOUT setting, establishes a per-export deadline that interacts with context cancellation during shutdown. However, no cross-language specification guarantees the reporting fate of dropped spans, and default timeout values and retry backoff behavior differ between Java, Go, Python and JavaScript runtimes. Teams deploying bounded shutdown contexts must therefore decide their own observability strategy for export loss.
31775 reputation · 16 Sept 2023, 20:06 UTC
Short answer: no OpenTelemetry SDK guarantees a countable, cross-language event when spans are dropped because an export deadline expires. Timeout-induced drops are handled inside the SDK/exporter boundary, and whether anything is surfaced — an error-handler callback, a metric, or only a debug log line — depends on the language and SDK version. If you need reliable accounting of export loss, you must add it yourself.
Confirmed, well-established behavior:
OTEL_EXPORTER_OTLP_TIMEOUT (or an API-level timeout/deadline option). The BatchSpanProcessor passes a context carrying that deadline to the exporter.forceFlush() and shutdown() honor context cancellation, so a bounded shutdown context can truncate the export window and cause the same kind of drop.Likely, but version-sensitive (verify against your exact SDK release):
otel.Handle error handler receives some exporter errors, but timeout-driven batch abandonment is not uniformly funneled there across languages.terminationGracePeriodSeconds shorter than the export timeout, or a shutdown context with a tight deadline, means the final flush is cancelled mid-export and the remaining buffer is dropped — silently, from your application's point of view.# Controlled shutdown test (conceptual, any SDK)
OTEL_EXPORTER_OTLP_TIMEOUT=2000 \
OTEL_BSP_SCHEDULE_DELAY=500 \
./your-app &
PID=$!; sleep 5; kill -TERM $PID
# Compare: spans produced (app log) vs spans received (collector debug exporter)If the gap matters for compliance or SLOs, do not rely on SDK-internal logging — instrument the loss yourself. One missing detail that would sharpen this answer: which SDK language and version you are running, since the exact metric names and error-handler behavior are version-specific and should be confirmed against that release's changelog.
Use comments to ask for clarification. Post a solution as an answer.
No question comments on this page.