Diagnosing OpenTelemetry Collector OTLP Exporter Failures

Problem: Zero telemetry at the backend

You deployed the OpenTelemetry Collector with an OTLP exporter, but the backend (e.g., Tempo, Jaeger, Prometheus, or a vendor SaaS) shows no incoming spans or metrics. Collector logs may contain context deadline exceeded, connection refused, rpc error: code = Unauthenticated, or 413 Request Entity Too Large. This guide walks through the most common root causes, ordered checks, and targeted fixes.

Symptom → Likely Cause Quick Reference

Observed Symptom	Typical Cause
Collector logs context deadline exceeded or connection refused	Wrong endpoint host/port, network policy blocking egress, or TLS handshake failure (backend expects mTLS, Collector sends plaintext)
rpc error: code = Unauthenticated (gRPC) or HTTP 401	Missing or malformed Authorization header (Bearer token) / API key; gRPC uses metadata, HTTP uses headers
High Collector memory/CPU, growing queue_size metric	Batch processor mis‑configured (send_batch_size > queue_size) or backend back‑pressure not handled; retry back‑off too aggressive
Spans arrive but missing attributes / resources	Resource detectors not enabled (k8s, cloud, host) or attribute processor dropping keys
Metrics temporality mismatch (cumulative vs delta)	OTLP exporter defaults to cumulative for Prometheus‑compatible backends; backend expects delta
Intermittent HTTP 413 Request Entity Too Large	Batch size exceeds backend limit; compression not enabled
No data despite healthy logs	Protocol mismatch — Collector uses OTLP/gRPC (port 4317) but backend listens on OTLP/HTTP (port 4318) or legacy Jaeger/Zipkin ports
TLS error x509: certificate signed by unknown authority	Self‑signed certs without CA trust; need tls.insecure_skip_verify (dev) or proper ca_file, cert_file, key_file for mTLS

Ordered Diagnostic Checks

1. Validate Collector configuration syntax
   Run on the host where the Collector binary lives (requires read access to the config file):

   otelcol-contrib --config=collector.yaml --dry-run

   Expected: exit code 0, no “unknown component” errors. If it fails, fix YAML references before proceeding.
2. Confirm endpoint reachability
   gRPC (port 4317) – from the Collector host (needs network egress):

   grpcurl -plaintext <backend-host>:4317 opentelemetry.proto.collector.trace.v1.TraceService/Export

   HTTP (port 4318) – same host:

   curl -v -X POST http://<backend-host>:4318/v1/traces \
     -H 'Content-Type: application/json' -d '{}'

   A 200/202 response means the endpoint is alive; connection refused or timeout points to network policy or wrong host/port.
3. Inspect Collector internal metrics for export failures
   Enable the Prometheus scrape endpoint (default :8888/metrics) and run:

   curl localhost:8888/metrics | grep otelcol_exporter_send_failed_spans_total

   A non‑zero counter confirms the exporter is attempting sends but failing.
4. Enable the debug exporter temporarily
   Add to collector.yaml:

   exporters:
     debug:
       verbosity: detailed
   service:
     pipelines:
       traces:
         exporters: [otlp, debug]
       metrics:
         exporters: [otlp, debug]

   Restart Collector (requires process restart permission). Logs will now show full payloads — verify that spans/metrics are being generated and that headers/metadata contain the expected Authorization token.
5. Check resource attribute presence
   Query the backend for service.name, cloud.provider, k8s.pod.name. Missing values indicate resource detectors (k8s, cloud, host) are not enabled or lack RBAC permissions (k8s detector needs pod metadata read).

Fixes Mapped to Findings

Network / Endpoint

• Correct exporters.otlp.endpoint to match backend scheme and port (e.g., https://tempo.example.com:4317 for gRPC, https://tempo.example.com:4318 for HTTP). Do not mix schemes in a single exporter instance.
• Update Kubernetes NetworkPolicy or cloud security groups to allow egress from Collector pods to the backend CIDR/port.

TLS & mTLS

• For development with self‑signed certs:

  exporters:
    otlp:
      endpoint: "https://backend:4317"
      tls:
        insecure_skip_verify: true

• Production mTLS — supply CA and client cert/key:

  tls:
    ca_file: "/etc/otel/certs/ca.pem"
    cert_file: "/etc/otel/certs/client.pem"
    key_file: "/etc/otel/certs/client-key.pem"

  Ensure files are readable by the Collector user (typically otelcol).

Authentication

• gRPC — add metadata via auth extension or static headers:

  exporters:
    otlp:
      endpoint: "https://backend:4317"
      headers:
        authorization: "Bearer ${API_TOKEN}"

• HTTP — same header map works; ensure the token is not wrapped in extra quotes.
• Validate the token works with a manual curl using the same header.

Batching & Back‑pressure

• Set send_batch_size ≤ queue_size (default queue 5000). Example safe config:

  processors:
    batch:
      timeout: 10s
      send_batch_size: 1000
      send_batch_max_size: 1000
      queue_size: 5000

• If backend returns 429/503, increase retry_on_failure back‑off:

  exporters:
    otlp:
      retry_on_failure:
        enabled: true
        initial_interval: 5s
        max_interval: 30s
        max_elapsed_time: 120s

• Enable gzip compression for HTTP to reduce payload size:

  exporters:
    otlp:
      compression: gzip

Resource & Attribute Detection

• Enable detectors in service.resource (order matters — later detectors overwrite earlier):

  service:
    resource:
      detectors: [env, k8s, cloud, host]
      attributes:
        - key: "deployment.environment"
          value: "production"
          action: upsert

• Grant the Collector ServiceAccount get, list, watch on pods for the k8s detector.

Metrics Temporality

• If backend expects delta (e.g., Prometheus remote write), set temporality explicitly:

  exporters:
    otlp:
      metric_exporter:
        prometheus:
          add_metric_suffixes: false
          temporality: delta

• Note: add_metric_suffixes: true appends _total, _bucket — breaks dashboards expecting base names.

Escalation Criteria

• After completing all checks, if otelcol_exporter_send_failed_spans_total continues to increase and debug exporter shows well‑formed payloads, engage the backend vendor — provide a capture of the debug logs and the exact Collector version (otelcol-contrib --version).
• If Collector memory grows unbounded despite correct batch settings, profile the process (go tool pprof) and consider upgrading to a newer Collector minor version (experimental components may leak).
• Persistent TLS handshake failures with valid certs may indicate cipher suite mismatch; verify backend TLS config and Collector Go version (Go 1.22+ defaults).

Verification After Fixes

Re‑run the diagnostic checks in order. Success indicators:

• otelcol_exporter_send_failed_spans_total stays at zero for >5 minutes.
• Backend shows increasing span/metric counts with expected resource attributes.
• Collector logs no longer contain deadline, auth, or 413 errors.

Limitations

Configuration validation (--dry-run) only checks syntax and component existence; it cannot detect runtime network policies, certificate trust chains, or backend‑side rate limits. The debug exporter adds log volume — disable it in production after verification. Batch processor mis‑configuration does not emit a startup warning; the send_batch_size ≤ queue_size rule must be enforced manually.

Version Assumptions

Guidance applies to OpenTelemetry Collector Contrib v0.100+ (Go 1.22). Behavior of experimental components (e.g., spanmetrics connector) may change across minor releases — pin the Collector image digest in production.