Enable OpenTelemetry Distributed Tracing in Ballerina HTTP Services

Desired Outcome

By the end of this guide you will have a Ballerina HTTP service that automatically creates OpenTelemetry spans for every incoming request, outgoing HTTP client call, and SQL operation. The spans will be exported to an OTLP endpoint (e.g., Jaeger) and visible in the Jaeger UI, giving you end‑to‑end distributed tracing without writing any instrumentation code.

Prerequisites

• Ballerina 2201.0‑LTS or newer installed (the observability module is not available in earlier releases).
• Access to an OTLP‑compatible collector or tracing backend such as Jaeger or Zipkin.
• Basic knowledge of HTTP services in Ballerina (a listener and a service definition).
• Network connectivity from the Ballerina process to the OTLP endpoint.

Step 1 – Upgrade Ballerina (if necessary)

If you are running a version older than 2201.0, upgrade first:

bal update

Verify the version with:

bal version

Step 2 – Import the Observability Module

Add the module to your Ballerina project. In the root of your project, create or edit Ballerina.toml:

[project]
org = "example"
name = "tracing-demo"
version = "0.1.0"

[build-options]
observability = true

[platform]
min-version = "2201.0"

[dependencies]
ballerina/observability = "@2201.0.0"

Run bal build to fetch the dependency.

Step 3 – Configure the OTLP Exporter

OpenTelemetry uses the OTLP protocol to send spans. Create a file trace-config.json (or embed the JSON inline) that defines the exporter. Example for Jaeger running locally on port 4317 (gRPC):

{
  "exporter": {
    "type": "otlp",
    "endpoint": "localhost:4317",
    "protocol": "grpc"
  },
  "sampling": {
    "ratio": 1.0
  }
}

In your service file, import the module and reference the configuration:

import ballerina/observability;

@observability:TracerConfig {
  configFile: "trace-config.json"
}
service helloService {
    resource function get hello() returns string {
        return "Hello, tracing!";
    }
}

Alternatively, you can set the configuration programmatically:

import ballerina/observability;

observability:TracerConfiguration tracerCfg = {
    exporter = {
        type = "otlp",
        endpoint = "localhost:4317",
        protocol = "grpc"
    },
    sampling = {
        ratio = 1.0
    }
};

observability:TracerConfig { config: tracerCfg }

Step 4 – Annotate the Service (Optional)

The default configuration automatically instruments HTTP listeners, client calls, and SQL operations. If you want to change the service name or add custom attributes, use the annotation:

@observability:TracerConfig {
  serviceName = "greeting-service",
  attributes = {"env" : "dev"}
}
service greetingService { /* ... */ }

Step 5 – Run the Service

Start the service normally:

bal run

Verify that the process outputs a line similar to:

[INFO] Observability: Tracer started, exporting to localhost:4317

Step 6 – Verify Traces in Jaeger

1. Launch Jaeger locally (if not already running):

docker run -d --name jaeger-all-in-one \
  -e COLLECTOR_OTLP_ENABLED=true \
  -p 16686:16686 -p 4317:4317 jaegertracing/all-in-one:latest

2. Send a request:

curl http://localhost:9090/hello

3. Open http://localhost:16686/ and search for greeting-service. You should see a trace with two spans: listener: get /hello and response: get /hello.

Cross‑Service Context Propagation

To confirm that trace context propagates, add an HTTP client call to another Ballerina service:

import ballerina/http;

http:Client client = new("http://localhost:9091/other");

resource function get hello() returns string {
    var resp = client->get("/");
    return resp.toString();
}

Run both services, call the first one, and verify that the Jaeger UI shows a single trace spanning both services.

Step 7 – Tune Sampling and Performance

High traffic can flood your backend with spans. Adjust the sampling ratio in the configuration:

{
  "sampling": {"ratio": 0.1}
}

Alternatively, use a probabilistic or traceidratio sampler if available in the future releases.

Expected Checks

• Service starts without error and logs the tracer startup message.
• Jaeger UI shows a trace for each request, including child spans for client calls.
• Trace IDs in the HTTP response headers contain the traceparent field (W3C).
• No dropped spans: if the OTLP endpoint is unreachable, check logs for warnings and retry.

Recovery Options

• Exporter unreachable: Ensure the OTLP endpoint is reachable on the specified host/port. Check firewall rules and container networking if using Docker.
• Spans missing: Verify the sampling ratio is not set to 0.0 and that the service name matches the Jaeger UI filter.
• Performance impact: Reduce the sampling ratio or enable batch export mode in the configuration to lower CPU usage.
• To disable tracing temporarily, remove the @observability:TracerConfig annotation or set enabled = false in the configuration.

Limitations

• Observability module is only available in Ballerina 2201.0‑LTS and newer.
• Spans are dropped silently if the OTLP exporter fails; enable logging to detect this.
• Current version does not support custom span creation via code; only automatic instrumentation is available.

Conclusion

With minimal configuration, Ballerina’s observability module gives you full OpenTelemetry tracing for HTTP services. The automatic instrumentation and simple OTLP exporter setup let you focus on business logic while gaining deep visibility into request flows.