Resilient OpenTelemetry Collector Pipeline with Batching, Retry and Dead Letter Handling

Telemetry disappears silently when an exporter backs off during a backend outage. The practical fix is a collector pipeline that ingests OTLP data, batches it, retries with backoff, and routes permanently failed items to a dead letter sink so data loss is visible and recoverable.

Desired outcome

A production collector that receives traces, metrics and logs via OTLP, delivers them reliably to a backend, and surfaces failures instead of dropping them. Latency is bounded by batch settings, and failures are observable via collector internal metrics and a dead letter store.

Prerequisites

An OpenTelemetry Collector deployment you can update, network reachability to an OTLP compatible backend, and consistent resource attributes and semantic conventions across services. You need permission to modify collector config and read its metrics endpoint. Collector config schema is version sensitive; processor names and exporter defaults change across major releases, so pin the collector version you test against.

Pipeline design

Receiver → Batch → Exporter with retry → Dead letter fallback. Batch reduces request overhead. Exporter retry with backoff protects transient failures. Dead letter captures items that exhaust retries.

Receiver and service definition

Expose OTLP gRPC and HTTP for all three signals on the same collector. Keep receivers separate per signal for clarity.

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch/traces:
    timeout: 5s
    send_batch_size: 512
  batch/metrics:
    timeout: 5s
    send_batch_size: 1024
  batch/logs:
    timeout: 5s
    send_batch_size: 512

Batch timeout controls maximum wait before flushing. Send batch size caps items per request. Larger batches improve throughput but increase tail latency and memory.

Exporter with retry and timeout

Configure the OTLP exporter with explicit timeout and retry. Retry is per exporter, not a separate processor.

exporters:
  otlphttp/backend:
    endpoint: http://<backend-endpoint>:4318/v1/traces
    timeout: 10s
    retry_on_failure:
      enabled: true
      initial_interval: 1s
      max_interval: 30s
      max_elapsed_time: 300s

Replace <backend-endpoint> with your backend host. Initial interval and max interval control backoff. Max elapsed time caps total retry duration. After expiry the item is considered failed.

Dead letter handling

For items that exhaust retries, route to a persistent sink. The collector can write failed items to a file exporter for manual re-injection. Dead letter storage requires lifecycle management to avoid unbounded disk growth.

exporters:
  file/deadletter:
    path: /var/lib/otel/deadletter/%Y-%m-%d.jsonl

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch/traces]
      exporters: [otlphttp/backend, file/deadletter]

Order matters. The primary exporter is tried first. On permanent failure the item can be sent to the dead letter exporter. Verify your collector version supports the dead letter routing pattern you use.

Expected checks

Verify collector starts without config errors and exposes health. Check /metrics for collector internal metrics such as queue size, export success and failure counters, and retry attempts. Confirm sample telemetry from a test service appears in the backend with expected service.name, resource attributes and timestamps.

Simulate a backend outage by blocking egress to <backend-endpoint> from the collector pod. Monitor export error counters and retry attempts. When the backend recovers, confirm backoff resumes and items drain.

Force a permanent failure by pointing the exporter to an invalid endpoint with max_elapsed_time short. Inspect /var/lib/otel/deadletter for captured items and confirm they contain trace_id and timestamp fields.

Recovery options

If latency is too high, reduce batch timeout or send_batch_size. If export errors persist, increase max_concurrent_requests on the exporter and verify network reachability and TLS settings.

For sustained backend issues, increase max_elapsed_time to allow longer retry windows, or route to an alternate exporter temporarily. Roll back collector config via versioned manifests. Keep previous config in git and apply with kubectl rollout undo or equivalent for your deployment method.

Limitations: batching and retry add latency. Semantic conventions and resource detection differ between language SDKs and versions, causing inconsistent attribute names in the backend. Dead letter files grow without rotation and retention policy.