Configure OpenTelemetry Collector to Ingest OTLP and Export to Jaeger, Prometheus, and Loki

Desired Outcome

Run a single OpenTelemetry Collector instance that:

• Accepts OTLP traces and metrics over gRPC (port 4317) and HTTP (port 4318).
• Exports traces to a Jaeger collector (Thrift Compact on 14268).
• Exposes metrics in a Prometheus‑scrapeable format on 8889.
• Pushes logs to Loki via HTTP on 3100.
• Applies per‑signal batching, memory limits, and enriches data with service.name and deployment.environment attributes.

Prerequisites

• Linux/macOS/Windows host with Docker 24+ or a native binary of otelcol‑contrib (v0.112.0 or later).
• Network connectivity to Jaeger, Prometheus, and Loki endpoints (or local containers).
• Optional TLS certificates if any backend requires mTLS.
• Environment variables OTEL_SERVICE_NAME and OTEL_DEPLOYMENT_ENVIRONMENT set for resource attribution, or define them directly in the config.

Configuration Overview

The Collector configuration is split into three logical parts:

• Receivers – where data enters.
• Processors – transform or buffer data.
• Exporters – push data to downstream systems.

Each signal (traces, metrics, logs) has its own pipeline, allowing independent batching and retry settings.

Step‑by‑Step Setup

1. Create config.yaml
   Use the following template. Replace placeholder values if your backends use different ports or require TLS.

2. # config.yaml
   receivers:
     otlp:
       protocols:
         grpc:
           endpoint: "0.0.0.0:4317"
         http:
           endpoint: "0.0.0.0:4318"
   
   processors:
     memory_limiter:
       limit_mib: 512
       spike_limit_mib: 128
       check_interval: 10s
     batch:
       timeout: 5s
       send_batch_size: 1024
       send_batch_max_size: 8192
     resource:
       attributes:
         service.name: "${OTEL_SERVICE_NAME:default-service}"
         deployment.environment: "${OTEL_DEPLOYMENT_ENVIRONMENT:dev}"
   
   exporters:
     jaeger:
       endpoint: "jaeger:14268"
       tls:
         insecure: true
     prometheus:
       endpoint: "0.0.0.0:8889"
     loki:
       endpoint: "http://loki:3100/loki/api/v1/push"
   
   service:
     extensions: [health_check]
     pipelines:
       traces:
         receivers: [otlp]
         processors: [memory_limiter, batch, resource]
         exporters: [jaeger]
       metrics:
         receivers: [otlp]
         processors: [memory_limiter, batch, resource]
         exporters: [prometheus]
       logs:
         receivers: [otlp]
         processors: [memory_limiter, batch, resource]
         exporters: [loki]
   

3. Validate the config
   Run a dry‑run to catch syntax errors before deployment.

4. otelcol‑contrib --config=config.yaml --dry-run
   

5. Run the Collector in Docker
   Map the required ports and mount the config file.

6. docker run \\
     --name otelcol \\
     -p 4317:4317 \\
     -p 4318:4318 \\
     -p 8889:8889 \\
     -p 3100:3100 \\
     -p 13133:13133 \\
     -v $(pwd)/config.yaml:/etc/otelcol/config.yaml \\
     otel/opentelemetry-collector-contrib:0.112.0 \\
     --config=/etc/otelcol/config.yaml
   

7. Verify that the collector is healthy
   The Collector exposes a /health_check endpoint on port 13133.

8. curl -s -o /dev/null -w "%{http_code}\n" http://localhost:13133
   # Expected output: 200
   

Validation Checks

• Prometheus metrics – Scrape http://localhost:8889/metrics from a Prometheus instance or curl and look for counters such as otelcol_receiver_accepted_spans_total increasing.
• Jaeger UI – Open http://localhost:16686 (or your Jaeger host) and search for the service name you set. Traces should appear with the correct span kind and attributes.
• Loki LogQL – In Grafana or Loki’s API, run:

  {service_name="${OTEL_SERVICE_NAME}"} |= "test-span"
  

  You should see log lines containing the trace ID.
• Collector logs – Inspect Docker logs for any ERROR or WARN messages. Example:

  docker logs otelcol
  

Recovery Options

• Backends unavailable – The batch processor buffers data in memory. If a backend goes down, the Collector will retry until the buffer reaches send_batch_max_size. To avoid data loss, monitor the otelcol_batcher_queue_size metric.
• Memory pressure – The memory_limiter pauses ingestion when memory usage exceeds limit_mib. Logs will show a warning. Increase the limit or adjust the container’s memory quota if you see frequent pauses.
• Configuration errors – Use --dry-run before redeploying. If the Collector exits immediately, the logs will contain a JSON error describing the misconfiguration.
• Service restart – Restart the Docker container to drain any queued data. The Collector will resume normal operation once the backends are reachable.

Limitations & Next Steps

• The example uses Jaeger’s Thrift Compact exporter, which lacks TLS support in Collector <0.110. If you need mTLS, switch to otlphttp exporter or upgrade Jaeger.
• Prometheus exporter shares the same port as the OTLP HTTP receiver if not separated. In production, bind each to a distinct service or use host networking with care.
• The resource processor only sets static attributes. For dynamic environments (Kubernetes, ECS), enable the k8s_observer or env processors to auto‑populate fields.
• Batch size and timeout are tuned for moderate traffic. For bursty workloads, increase send_batch_size or reduce timeout to lower latency.
• Always run health checks and monitor the Collector’s own metrics to detect backpressure or failures early.

Conclusion

With a single otelcol-contrib instance you can ingest OTLP traces, metrics, and logs, enrich them with resource attributes, and forward them to Jaeger, Prometheus, and Loki. The modular pipeline lets you adjust batching, retries, and memory limits per signal, giving you fine‑grained control over latency and reliability. Follow the steps above, validate each component, and you’ll have a robust observability foundation in minutes.