Configure the OpenTelemetry OTLP/gRPC Exporter for Trace Export

Desired outcome

Send trace data from an instrumented application to an observability backend using the OpenTelemetry OTLP/gRPC exporter. After configuration, spans should be transmitted reliably, appear in the backend UI, and carry the expected resource attributes (e.g., service.name).

Prerequisites

• An OpenTelemetry SDK already initialized in the application (tracer provider created).
• A reachable gRPC endpoint that accepts OTLP traces (default localhost:4317) – this can be a collector, Jaeger, Tempo, or any OTLP‑compatible receiver.
• If the endpoint uses TLS, the application must trust the server’s certificate (e.g., via the JVM trust store or system CA bundle).
• Matching versions of the OpenTelemetry SDK and the OTLP exporter library (e.g., both 1.28.0 or newer) to avoid silent span drops.

Procedure

1. Add the OTLP/gRPC exporter dependency

   For a Java Maven project, add the following to pom.xml:

   <dependency>
       <groupId>io.opentelemetry</groupId>
       <artifactId>opentelemetry-exporter-otlp</artifactId>
       <version>1.28.0</version>
       </dependency>
   

   Equivalent coordinates exist for Gradle, Go, Python, Node.js, and .NET; consult the SDK documentation for the exact artifact.

2. Create the exporter instance

   Instantiate the OTLP/gRPC exporter, pointing it at the collector endpoint. Example in Java:

   import io.opentelemetry.exporter.otlp.trace.OtlpGrpcSpanExporter;
   import io.opentelemetry.sdk.trace.SpanProcessor;
   import io.opentelemetry.sdk.trace.export.BatchSpanProcessor;
   
   OtlpGrpcSpanExporter otlpExporter = OtlpGrpcSpanExporter.builder()
           .setEndpoint("http://collector.example.com:4317")   // use https:// for TLS
           .setTimeout(java.time.Duration.ofSeconds(10))
           .build();
   

   If TLS is required and uses a custom trust store, configure the underlying Netty channel via .setTrustCertificates(...) or JVM system properties.

3. Register the exporter with a tracer provider

   Wrap the exporter in a BatchSpanProcessor to gain asynchronous, retry‑enabled export:

   SpanProcessor spanProcessor = BatchSpanProcessor.builder(otlpExporter)
           .setMaxQueueSize(2048)          // tune based on expected traffic
           .setScheduleDelay(java.time.Duration.ofSeconds(5))
           .setExportTimeout(java.time.Duration.ofSeconds(30))
           .build();
   
   // Assuming `tracerProvider` is already created:
   tracerProvider.addSpanProcessor(spanProcessor);
   

4. Start the application and generate traffic

   Run the service as usual. Exercise code paths that create spans (e.g., HTTP requests, database calls). The exporter will attempt to send batches to the gRPC endpoint.

Expected checks

• No export errors in logs: Enable OpenTelemetry debug logging (OTEL_LOG_LEVEL=debug or via SDK configuration) and look for lines such as "Exporting traces" followed by a successful response code.
• Spans appear in the backend UI: In Jaeger, Tempo, or your observability platform, search for traces with the service.name resource attribute you set (e.g., my‑order‑service).
• Resource attributes are present: Open a trace and verify that attributes like service.name, service.version, and any custom attributes you added are attached to each span.
• Collector/receiver shows incoming data: If you run a temporary collector with otelcol-contrib --config=otlp-receiver.yaml, its console output should list received spans.

Recovery options

• Exporter retries: The BatchSpanProcessor uses exponential backoff; transient network issues will trigger automatic retries.
• Fallback to OTLP/HTTP: If gRPC is blocked, replace OtlpGrpcSpanExporter with OtlpHttpSpanExporter pointing to the same endpoint’s /v1/traces path.
• Logging exporter as a safety net: Add a SimpleSpanProcessor with a ConsoleSpanExporter or LoggingSpanExporter to capture spans locally when the OTLP exporter fails repeatedly.
• Adjust batch processor settings: Increase maxQueueSize or decrease scheduleDelay to relieve memory pressure under high load.

Limitations and practical verification

• Version mismatch: Using an exporter newer than the SDK can cause silent drops. Verify versions with mvn dependency:tree (Java) or the equivalent tool for your language.
• TLS trust failures: If the collector uses a self‑signed certificate, add it to the application’s trust store or disable verification only in test environments (OTEL_EXPORTER_OTLP_TRUST_CERTIFICATES=...).
• High‑volume tuning: Monitor JVM heap or process memory; if you see OOM or growing queues, reduce maxQueueSize or increase the number of worker threads via the collector’s receiver settings.
• Practical verification step: After deploying the change, run:

OTEL_LOG_LEVEL=debug java -jar myapp.jar 2>&1 | grep -i "exporting traces"

You should see one or more lines ending with a status like "OK" or "200". Simultaneously, check the backend UI for new traces within a few seconds of generating load.