Isolating Microservice Latency with New Relic Custom Span Attributes

The Problem: Structural Traces Lack Business Context

When a microservice experiences a latency spike, a standard distributed trace shows you the structural path: Service A called Service B, which queried Database C. While this identifies where the delay happened, it rarely explains why. You might see that a specific database query is slow, but you cannot tell if that slowness is universal or isolated to a specific high-volume tenant, a particular API version, or a user with a massive data set.

Thesis: Use Custom Attributes to Bridge the Gap Between Infrastructure and Business Logic

New Relic agents automatically capture the "skeleton" of a request. To make these traces actionable, you must enrich them with custom attributes—key-value pairs that provide business context. By attaching identifiers like tenant_id or feature_flag_status to spans, you transform a generic latency spike into a searchable signal, allowing you to group and filter traces by the actual business dimensions causing the bottleneck.

Implementing Custom Attributes

To add context, you use the New Relic SDK for your specific language. It is important to distinguish between Transaction attributes (which describe the overall request) and Span attributes (which describe a specific segment of work). For distributed tracing, you want the attribute attached to the span to ensure the context travels with the request across service boundaries.

Example: Adding Context in Node.js

In a Node.js environment using the New Relic agent, you can use the addCustomAttribute method. The following example demonstrates adding a tenant identifier to the current transaction, which the agent can then propagate to associated spans depending on your configuration.

const newrelic = require('newrelic');

app.get('/api/data', (req, res) => {
    const tenantId = req.headers['x-tenant-id'] || 'unknown';
    
    // Attach the tenant ID to the current transaction
    newrelic.addCustomAttribute('tenantId', tenantId);
    
    // Business logic here...
    res.send('Data retrieved');
});

Execution Details: Run this within your application code. No special permissions are required beyond the standard agent installation. Replace 'x-tenant-id' with your actual header name. Risk: If the agent is not initialized before this call, the method may be undefined or fail silently.

Verifying and Querying the Data

Before building dashboards, you must verify that the attributes are being ingested and identify exactly how they are stored in the New Relic database, as naming and nesting can vary by agent version.

Step 1: Visual Verification

1. Generate traffic with distinct attribute values (e.g., tenantId = 'alpha' and tenantId = 'beta').
2. Navigate to APM > Distributed Tracing in the New Relic UI.
3. Select a specific trace and click on an individual span. Check the Span details panel to see if your custom attribute is listed.

Step 2: NRQL Validation

Because attribute mapping can differ, run a keyset() query to see the exact field names available on your Span events:

SELECT keyset() FROM Span LIMIT 1

Once you confirm the field name (e.g., tenantId), you can isolate latency for a specific group:

SELECT average(duration) FROM Span 
WHERE tenantId = 'alpha' 
FACET name 
SINCE 1 hour ago

Note: Always verify the unit of duration (seconds vs milliseconds) by running a sample query before setting alert thresholds.

Trade-offs: Cardinality and Sampling

While adding attributes is powerful, it introduces two primary technical constraints:

• Cardinality: Avoid using unique IDs (like request_id or user_uuid) as custom attributes. High-cardinality data—where almost every event has a unique value—can degrade query performance and, in some account tiers, may impact data ingestion limits. Stick to categories like plan_level, region, or tenant_id.
• Sampling: New Relic does not keep every single trace to save overhead. If you are looking for a "needle in a haystack" (a single slow request for one specific user), the sampling algorithm might have dropped that trace. To mitigate this, use Error-based sampling or increase the sampling rate for critical endpoints.

Actionable Summary

To stop guessing why your microservices are slow, follow this workflow: instrument your request entry points with low-cardinality business identifiers using the New Relic SDK, verify the attribute's presence in the Trace View, and use SELECT keyset() to confirm the exact NRQL field name. Once verified, build NRQL alerts that trigger when latency exceeds your SLO for a specific tenant or feature flag, ensuring you spend your debugging time on the actual root cause rather than hunting through generic traces.