Designing an OpenTelemetry Collector OTLP/HTTP Exporter with Mutual TLS and Retry Logic

Requirements

Export trace and metric data from an OpenTelemetry Collector to a backend over HTTP with transport security, resilience to transient network issues, and at-least-once delivery. The concrete requirements are:

• Mutual TLS (mTLS) so both the Collector and the backend prove their identities.
• Configurable endpoint URL.
• Batch sending to amortize HTTP overhead.
• Retry-on-failure with exponential backoff for temporary glitches.
• Visibility into success and failure through metrics and health endpoints.

Smallest suitable design

A single OTLP/HTTP exporter component inside the Collector satisfies these requirements. The design combines the OTLP/HTTP protocol, TLS client authentication (mTLS), a batch processor, and an exponential backoff retry mechanism.

Below is a minimal Collector configuration snippet (YAML). Adjust paths and values to match your environment.

exporters:
  otlphttp:
    endpoint: https://telemetry.example.com/v1/traces
    tls:
      ca_file: /etc/otel/certs/ca.pem
      cert_file: /etc/otel/certs/client.pem
      key_file: /etc/otel/certs/client-key.pem
    send_batch_size: 512
    timeout: 10s
    retry_on_failure:
      enabled: true
      initial_interval: 5s
      max_interval: 60s
      max_elapsed_time: 300s

Note that mTLS support for the OTLP/HTTP exporter was added in Collector release v0.70. Older versions lack this feature and will fail to start if mTLS is configured.

Trust and data boundaries

The exporter operates inside the Collector's trust boundary. Data is treated as trusted once a Receiver has accepted it and passed it to the exporter. The mTLS handshake establishes a trusted channel before any telemetry leaves the process; externally, data remains untrusted until the TLS handshake succeeds and the backend validates the client certificate. A failed handshake means no data leaves the Collector.

Operational checks

Monitor the following metrics, typically exposed via the Collector's internal Prometheus endpoint:

• otelcol_exporter_sent_spans – counter of spans successfully handed off to the HTTP client.
• otelcol_exporter_failed_spans – counter of spans that failed after all retries.
• otelcol_exporter_send_queue_length – current number of batches waiting to be sent.
• otelcol_exporter_tls_handshake_errors – increments each time the mTLS handshake fails.

Alert on sustained high retry counts or authentication failures. The Collector's health endpoint also reports exporter status; a non-OK state indicates configuration or connectivity problems.

Failure modes

• Network interruption – the HTTP POST fails and the batch is re-queued. If the queue exceeds its limit, newer batches are dropped.
• mTLS handshake failure – logged as an authentication error, otelcol_exporter_tls_handshake_errors increments, and the batch is discarded.
• Expired or revoked client certificate – each attempt triggers a handshake error, causing a continuous retry loop without progress. This may remain silent unless the exporter's error metrics are monitored.
• Excessive batch size – a large send_batch_size can cause memory spikes under burst traffic. Size should be tuned based on observed average payload and available heap.

Conditions that would change the design

The current design assumes a push model, HTTP transport, and at-least-once semantics. Any of the following would require a different approach:

• Pull-based telemetry – if the backend scrapes the Collector, a Receiver (for example, Prometheus) would replace the exporter, and mTLS would be configured on the Receiver side.
• Adopting gRPC OTLP – for binary efficiency, switch to the otlp exporter with gRPC. The retry and batching logic remains similar, but protocol-specific settings change.
• Strict exactly-once delivery – would need a durable storage queue and an acknowledgment protocol from the backend, moving beyond the in-memory batch retry.

Practical verification

1. Start the Collector with the configuration above and a simple tracer that generates a few spans.
2. Check the backend logs for successful client certificate verification.
3. Observe otelcol_exporter_sent_spans increase while otelcol_exporter_failed_spans stays near zero.
4. Introduce a temporary network block (for example, a firewall drop). Verify that otelcol_exporter_send_queue_length rises, and after the block is lifted, queued spans are resent and the queue length returns to baseline.
5. Replace the client certificate with an expired or invalid one. Confirm that the Collector logs TLS handshake errors, otelcol_exporter_failed_spans increases, and otelcol_exporter_sent_spans does not advance.