Designing a Minimal OpenTelemetry Sidecar Architecture for Kubernetes

The Problem: Trace Volume and Network Overhead

Sending telemetry directly from an application to a remote backend often introduces two primary issues: increased application latency due to network I/O and overwhelming the backend with a 100% trace volume that is rarely needed for debugging. To solve this, a local proxy is required to handle batching, sampling, and protocol translation without adding significant complexity to the application code.

The Smallest Suitable Design

The most efficient minimal design is the Collector Sidecar pattern. In this architecture, the OpenTelemetry (OTel) Collector runs as a secondary container within the same Kubernetes Pod as the application. This allows the application to send traces to localhost, minimizing network hops and avoiding the need for complex service discovery for the telemetry pipeline.

The pipeline consists of three stages:

• Receiver: OTLP (OpenTelemetry Protocol) over gRPC. This provides a high-performance binary format.
• Processor: A combination of a batch processor to reduce the number of outgoing requests and a probabilistic_sampler to drop a percentage of traces at the source.
• Exporter: A backend-specific exporter (e.g., Jaeger) that pushes the processed data to a centralized storage cluster.

Trust and Data Boundaries

By deploying the Collector as a sidecar, the trust boundary is confined to the Pod. The application communicates with the Collector over the loopback interface (127.0.0.1), meaning trace data never leaves the Pod in an unencrypted or unauthenticated state until it reaches the Collector's exporter.

Data Boundary Flow: Application (OTLP/gRPC) $\rightarrow$ Localhost $\rightarrow$ Collector (Processing/Sampling) $\rightarrow$ External Network $\rightarrow$ Backend.

Configuration Example

The following configuration defines a minimal pipeline. This should be applied as a ConfigMap and mounted into the Collector container. This example assumes OTel Collector v0.90.0 or later.

# otel-collector-config.yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317

processors:
  # Batching reduces the number of RPC calls to the backend
  batch:
    send_batch_size: 1024
    timeout: 1s
  # Probabilistic sampler keeps only 10% of traces
  probabilistic_sampler:
    sampling_percentage: 10.0

exporters:
  otlp/jaeger:
    endpoint: "jaeger-collector.monitoring.svc.cluster.local:4317"
    tls:
      insecure: true # Use secure TLS in production environments

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [probabilistic_sampler, batch]
      exporters: [otlp/jaeger]

Operational Checks and Failure Modes

To ensure the sidecar is functioning, implement the following checks:

• Health Probes: Configure Kubernetes livenessProbe and readinessProbe to hit the Collector's health check extension. If the Collector crashes, the Pod may be marked unhealthy, preventing the application from sending data into a "black hole."
• Resource Constraints: Sidecars consume CPU and Memory from the Pod's total quota. Monitor the otelcol_process_runtime_heap_alloc_bytes metric to ensure the batch processor isn't consuming excessive memory during spikes.

Failure Mode	Impact	Detection Method
Backend Unavailable	Collector buffers traces until memory limit is reached, then drops data.	Monitor otelcol_exporter_enqueue_failed_spans.
Sampler Misconfiguration	Too much data (cost/performance) or too little (loss of visibility).	Compare application span count vs. backend received count.
gRPC Connection Timeout	Application may block or throw errors depending on SDK configuration.	Check application logs for DeadlineExceeded or Unavailable.

Verification Steps

To verify the implementation, run the following steps from a terminal with kubectl access:

1. Connectivity Check: Exec into the application container and attempt to reach the Collector port: nc -zv localhost 4317.
2. Sampling Validation: Generate 100 known traces from the application. Query the Jaeger UI; if the probabilistic_sampler is set to 10%, you should see approximately 10 traces.
3. Traffic Analysis: Use tcpdump on the node to verify that gRPC compression is active if enabled, looking for the grpc-encoding: gzip header in the packets sent to the backend.

Design Evolution: When to Change

The sidecar model is ideal for low-to-medium scale. However, you should move to a DaemonSet (one Collector per node) or a Gateway (centralized cluster) if:

• Resource Overhead: The memory footprint of running 100+ Collector instances across a cluster exceeds the cost of the application itself.
• Complex Routing: You need to route traces to multiple backends based on complex attributes that require a global view of the traffic.
• Management Burden: Updating the Collector configuration requires restarting every application pod rather than a single telemetry fleet.