Scaling OpenTelemetry Pipelines: Batch Processor Meets Memory Limiter

The Problem: Telemetry Pipelines That Collapse Under Load

Your application is humming along, generating spans, metrics, and logs. Then traffic spikes. The OpenTelemetry Collector's memory grows until the OOM killer strikes, or the exporter starts returning 413 Payload Too Large errors. Neither is acceptable in production.

The fix isn't a single setting—it's a deliberate pairing: the batch processor to amortize export overhead, and the memory limiter to enforce a hard ceiling on memory usage. Together they let you scale throughput without sacrificing stability.

Why Order Matters: Memory Limiter First

Place the memory limiter before the batch processor in your pipeline. This ordering ensures backpressure propagates upstream before the batcher's internal buffers grow unbounded.

processors:
  memory_limiter:
    check_interval: 1s
    limit_mib: 512          # hard limit
    spike_limit_mib: 128    # soft limit (triggers pause)
    limit_percentage: 80    # alternative: percentage of available memory
  batch:
    timeout: 5s
    send_batch_size: 8192
    send_batch_max_size: 8192  # 8k spans ~ 1 MB for typical OTLP payloads
exporters:
  otlphttp:
    endpoint: "https://backend.example.com/v1/traces"
    compression: gzip
service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [otlphttp]

With this order, when memory_limiter hits its soft limit, it pauses the otlp receiver. Upstream SDKs (Java, Python, Node auto-instrumentation) then block or queue locally, giving you a natural backpressure signal instead of silent drops.

Sizing the Batch: Match Your Backend's Limits

The batch processor's send_batch_max_size must stay below your backend's maximum request payload. Common limits:

• Honeycomb, Grafana Cloud: 1 MB
• Datadog: 3.5 MB
• Tempo: 10 MB
• Generic OTLP/HTTP: often 4 MB (configurable)

Start at 80% of the backend limit. For a 1 MB backend, send_batch_max_size: 8192 spans (assuming ~120 bytes/span) keeps you near 1 MB. Monitor otelcol_processor_batch_batch_send_size and otelcol_exporter_send_failed_spans to tune.

The timeout (default 5s) trades latency for batch completeness. Lower it to 1–2s if you need faster visibility; raise it to 10s if your backend handles large batches efficiently and you want fewer requests.

Worked Example: Tuning a Kubernetes DaemonSet Collector

Scenario: A DaemonSet Collector receives OTLP/gRPC from node-local pods. Each pod's auto-instrumentation sends batches of up to 512 spans. The backend accepts 1 MB per request.

1. Set receiver limit: max_recv_msg_size_mib: 4 on the OTLP receiver so 512-span batches aren't silently dropped.
2. Configure memory limiter: limit_mib: 1024, spike_limit_mib: 256 on a 2 GB container limit (50% hard, 12.5% soft).
3. Configure batch: send_batch_max_size: 8192, timeout: 2s.
4. Deploy and observe: Enable the Collector's zPages (http://localhost:55679/debug/metrics) and scrape otelcol_processor_memory_limiter_dropped_spans, otelcol_receiver_refused_spans, and otelcol_processor_batch_batch_send_size into Prometheus.

If dropped_spans > 0 during normal traffic, raise limit_mib or add a second Collector instance. If refused_spans appears on the application side, the pause is cascading—consider increasing spike_limit_mib or reducing batch timeout to drain faster.

Trade-offs You'll Live With

• Pause vs. drop: Memory limiter's pause mode preserves data but adds latency to instrumented applications. Drop mode keeps the pipeline flowing but loses telemetry. Choose based on whether your SLOs prioritize completeness or responsiveness.
• No cross-batch ordering: The batch processor does not guarantee ordering across batches. If your backend requires strict ordering (rare), disable batching or use a single-threaded exporter.
• Shared Collector contention: Memory limits are process-wide. A noisy pipeline in a multi-tenant Collector can starve critical services. Isolate high-cardinality or high-volume services into dedicated Collector instances.

Verify Before You Ship

Run a local load test:

# Start Collector with your config
otelcol-contrib --config test.yaml

# Generate load (example using otel-cli)
for i in {1..10000}; do
  otel-cli span --name "test.op" --attribute "load.test=true" --endpoint localhost:4317
done

Watch the zPages metrics. Confirm batch_send_size clusters near your send_batch_max_size, memory_limiter_dropped_spans stays at zero, and receiver_refused_spans only appears during intentional backpressure tests.

Check your Collector version's release notes (v0.80+ for stable memory_limiter, v0.110+ for current schema) and your backend's documented ingestion limits before finalizing values.

Closing Checklist

• Memory limiter first in processor chain
• send_batch_max_size ≤ 80% of backend payload limit
• Receiver max_recv_msg_size_mib ≥ agent batch size
• Alert on otelcol_processor_memory_limiter_dropped_spans > 0
• Alert on otelcol_receiver_refused_spans sustained > 0
• Separate Collector instances for critical vs. noisy pipelines

This combination has been stable since 2022 and works identically across DaemonSet, sidecar, and standalone modes. The configuration is boring—but that's the point. Boring configurations don't wake you up at 3 AM.