Adding Custom Spans in Datadog APM to Trace Asynchronous Workflows

The problem: blind spots in async processing

When a request hands off work to a background worker or message queue, Datadog’s default APM instrumentation often sees only the HTTP entry and exit points. The time spent inside the worker, the delay on the queue, or any downstream calls appear as gaps, making it hard to pinpoint latency outliers.

Thesis: custom spans + proper context propagation close those gaps

By wrapping the specific logic you want to measure with a Datadog (or OpenTelemetry) span and ensuring the trace context travels with the message, you get a single, end‑to‑end trace that shows every piece of work, its duration, and any tags you add.

Section 1: Why custom spans matter for async gaps

Standard instrumentation creates spans at network boundaries (incoming HTTP, outgoing HTTP, database calls). Custom spans let you instrument arbitrary code blocks—such as a costly transformation, a lock acquisition, or a retry loop—without changing the service’s public interfaces.

Section 2: Implementing a custom span (Python example)

Assuming the Datadog ddtrace library is installed and the agent is running:

import ddtrace
from ddtrace import tracer

def process_order(order_id):
    # Start a custom span around the expensive validation step
    with tracer.trace('order.validation', resource='validate_order') as span:
        span.set_tag('order.id', str(order_id))
        # Simulate work
        validation_result = expensive_validation(order_id)
        span.set_tag('validation.result', validation_result)
    # Continue with normal flow (e.g., publish to queue)
    publish_to_queue(order_id, validation_result)

def expensive_validation(order_id):
    # placeholder for real logic
    return True

The tracer.trace context manager creates a span named order.validation. Any tags you set appear in the Datadog UI under that span.

Section 3: Propagating context across a message queue

To keep the trace alive, inject the trace IDs into the message headers before publishing and extract them on the consumer side.

import json
import pika
from ddtrace.propagation import http.HTTPPropagator

def publish_to_queue(order_id, payload):
    connection = pika.BlockingConnection(pika.ConnectionParameters('localhost'))
    channel = connection.channel()
    channel.queue_declare(queue='order_tasks')

    # Create a carrier dict for headers
    carrier = {}
    HTTPPropagator().inject(tracer.current_span(), carrier)

    # Attach trace context as message headers
    headers = {
        'trace_id': carrier.get('x-datadog-trace-id'),
        'parent_id': carrier.get('x-datadog-parent-id'),
        'sampling_priority': carrier.get('x-datadog-sampling-priority')
    }
    channel.basic_publish(
        exchange='',
        routing_key='order_tasks',
        body=json.dumps({'order_id': order_id, 'payload': payload}),
        properties=pika.BasicProperties(headers=headers)
    )
    connection.close()

On the worker:

def callback(ch, method, properties, body):
    msg = json.loads(body)
    carrier = {}
    if properties.headers:
        carrier.update({
            'x-datadog-trace-id': properties.headers.get('trace_id'),
            'x-datadog-parent-id': properties.headers.get('parent_id'),
            'x-datadog-sampling-priority': properties.headers.get('sampling_priority')
        })
    # Extract context to continue the same trace
    HTTPPropagator().extract(carrier)
    with tracer.trace('order.worker.process', resource='process_order') as span:
        span.set_tag('order.id', str(msg['order_id']))
        # do work...
    ch.basic_ack(delivery_tag=method.delivery_tag)

With extraction, the worker’s span becomes a child of the original order.validation span, preserving the trace ID across the queue.

Section 4: Verifying in Datadog and trade‑offs

After deploying the instrumented services:

• In the APM Traces list, look for a trace whose root span is order.validation (or whatever you named it).
• Click the trace to see the Trace Map or Flame Graph; you should see the custom span, the queue publish/consumer, and the worker’s processing span linked by the same trace ID.
• Check that tags you set (e.g., order.id) appear under the custom span.

Limitations / trade‑offs

• Each extra span adds a small CPU overhead; keep spans at a meaningful granularity (e.g., per‑logical step, not per line).
• High‑volume tracing can increase ingest costs; adjust the sample_rate or use Datadog’s priority sampling to control volume.
• If context propagation is missed (e.g., forgetting to inject headers), the worker will start a new, unrelated trace, fragmenting the view.

Practical way to check the result

After a test run, open the Trace Search UI, filter by service:your-service and name:order.validation. Verify that the trace contains exactly the spans you expect and that the trace.id is identical from the web request through the worker.

Actionable closing

Start small: add a single custom span around the slowest asynchronous step you suspect, propagate context across your queue, and confirm the trace appears intact. Once you see the benefit, gradually add more spans for other costly blocks, monitoring overhead and ingest usage along the way. This approach turns invisible async work into visible, actionable data in Datadog APM.