Question
OTLP Exporter Stops Sending Data When DNS Resolution Fails
Kavi Byte
0 reputation · 04 Mar 2023, 03:36 UTC
49.2K views1
Loading…
0 reputation · 04 Mar 2023, 03:36 UTC
The OpenTelemetry OTLP exporter resolves the collector endpoint via the system DNS resolver before establishing a TLS connection. When the hostname cannot be resolved, the exporter logs an error and ceases sending telemetry until the application is restarted. Likewise, a failed TLS handshake due to an invalid or self‑signed certificate results in an immediate stop of data transmission. The exporter offers no built‑in mechanism to override DNS resolution or to disable certificate validation, aside from external environment configuration. Consequently, there is uncertainty around how the exporter should handle transient DNS failures or certificate validation errors, and whether an automatic retry or an insecure mode should be supported.
To clarify the intended behavior, we need to address the following:
31775 reputation · 04 Mar 2023, 12:34 UTC
The OTLP exporter does not typically perform a one-time DNS resolution at startup; instead, it relies on the underlying runtime (gRPC or HTTP) to resolve the collector endpoint during connection establishment. When DNS resolution fails or a TLS handshake is rejected, the exporter treats these as transport-level errors. Depending on the SDK implementation, these errors trigger the internal retry policy. If the failures persist, the exporter's internal buffer may saturate, leading to backpressure and the appearance of a complete stop in data transmission.
Current OTLP SDKs generally follow these patterns regarding resolution and security:
opentelemetryOTEL_EXPORTER_OTLP_INSECURE=true for gRPC or providing a custom TLS configuration in the code).If your exporter has stopped sending data following a DNS or TLS event, follow these steps to restore connectivity:
nslookup [your-collector-endpoint]To provide specific retry intervals and timeout settings, please specify the OpenTelemetry SDK language (e.g., Java, Go, Python) and the transport protocol (gRPC or HTTP/protobuf) being used, as these significantly alter the connection pooling and retry behavior.
Use comments to ask for clarification. Post a solution as an answer.
31,775 reputation · 04 Mar 2023, 06:23 UTC
In practice, the gRPC‑based OTLP exporters rely on the underlying gRPC channel’s back‑off logic. A failed DNS lookup is treated as a connection error, so the channel will automatically re‑attempt the target name with exponential back‑off. The HTTP exporter follows the language HTTP client’s retry policy, which also retries transient connection errors but only up to a configurable retry budget.
OTEL_EXPORTER_OTLP_INSECURE=true turns off TLS entirely. To trust a self‑signed cert, supply a CA bundle via OTEL_EXPORTER_OTLP_CERTIFICATE.otel_exporter_queue_size or otel_exporter_retry_count) is the recommended way to detect silent failures.For transient DNS outages, a restart is usually unnecessary—the exporter will recover once the host resolves again. If you see persistent stops, verify the retry budget or add a fallback exporter (e.g., file‑based or a local collector).