New Relic APM Custom Attributes: Adding Business Context to Distributed Traces

The Problem: Traces Without Business Context

Distributed traces show latency and errors across services, but they don't tell you which customer, tenant, or feature flag triggered the request. Without that context, SRE and product teams can't correlate performance issues to business impact—like a slow checkout for enterprise tenants or a regression behind a feature flag.

Takeaway: Use the agent's request-scoped addCustomAttribute API to attach business identifiers (tenantId, requestId, feature flag) at the transaction level. These attributes propagate automatically with distributed tracing headers, appear on spans created after the call, and become filterable dimensions in NRQL and the APM UI.

How It Works: Request-Scoped Attribute API

Each New Relic language agent exposes a thread- or async-context-local API that binds attributes to the currently executing transaction. Call it early in the request lifecycle—ideally in middleware or a request filter—after you've extracted the identifier from headers, JWT claims, or routing parameters.

Worked Example: Node.js Express Middleware

// middleware/newrelic-tenant.js
const newrelic = require('newrelic');

function attachTenantContext(req, res, next) {
  // Extract tenant from header, JWT, or subdomain
  const tenantId = req.headers['x-tenant-id'] 
    || (req.user && req.user.tenantId)
    || 'unknown';

  // Attach to the current transaction
  if (newrelic.addCustomAttribute) {
    newrelic.addCustomAttribute('tenantId', tenantId);
  }

  // Optional: set a consistent transaction name for grouping
  if (newrelic.setTransactionName) {
    newrelic.setTransactionName(`${req.method} ${req.route?.path || req.path}`);
  }

  next();
}

module.exports = attachTenantContext;

Where to run: In your Express app, register this middleware before route handlers but after authentication middleware that populates req.user. Requires the newrelic npm package (v8+) and a valid license key in newrelic.js.

Expected check: After deployment, open a recent transaction trace in the APM UI → Attributes tab. You should see tenantId listed with the correct value.

Risk: Calling addCustomAttribute after the transaction ends (e.g., in an async callback that fires post-response) silently fails—the attribute won't appear.

Equivalent Patterns in Other Languages

• Java: NewRelic.addCustomParameter("tenantId", tenantId); in a servlet filter or Spring interceptor.
• Python: newrelic.agent.add_custom_attribute('tenantId', tenant_id) in Django middleware or FastAPI dependency.
• Go: txn := nrtxn.FromContext(ctx); txn.AddAttribute("tenantId", tenantId) using the nrtxn package.
• .NET: IAgent agent = Agent.Instance; agent.CurrentTransaction?.AddCustomAttribute("tenantId", tenantId); in an ASP.NET Core middleware.

All agents attach the attribute to the transaction event and to any spans created after the call. The attribute also propagates via W3C traceparent/tracestate headers to downstream services that have distributed tracing enabled.

Propagation in Distributed Traces

When service A calls service B over HTTP/gRPC, the New Relic agent injects trace headers that include custom attributes (subject to distributed_tracing.attributes.include configuration). Service B's agent reads those headers and recreates the attributes on its local transaction. This means tenantId set in your API gateway appears on database spans in a downstream payment service without extra code.

Configuration note: Ensure distributed_tracing.enabled: true (default since agent v7) and verify distributed_tracing.attributes.include isn't filtering out your keys. The default includes all custom attributes.

Limits and Operational Pitfalls

Cardinality and Ingest Cost

Each unique attribute value creates a new facet in NRDB. High-cardinality values—userId, sessionId, request UUID—explode timeseries cardinality and increase ingest bills. A tenantId with 500 values is fine; a userId with 50,000 daily active users is not.

Rule of thumb: Only attribute dimensions you'll actually facet on in dashboards or alerts. Use requestId for debugging individual traces, not for aggregation.

Per-Account Limits and Dropping

• Attribute count: ~255 attributes per event (transaction/span). Extras are dropped silently.
• Value length: 255 characters for transaction attributes, 4,000 for span attributes. Longer values are truncated.
• PII filtering: Agents automatically drop attributes matching patterns like password, ssn, credit_card. Custom regex can be added via attributes.exclude.
• Sampling: Transaction sampling (default 1000/min per host) means not every request is stored. Attributes on unsampled transactions are never ingested.

Common Mistakes

1. Adding attributes too late. Calling addCustomAttribute in a finally block, after res.send(), or in a background job loses the transaction context. Set it in the request entry middleware.
2. Inconsistent naming across services. Service A uses tenantId, service B uses tenant_id, service C uses customerId. Dashboards and NRQL facets break. Define a naming convention (e.g., camelCase, prefix with nr.) and enforce it in shared libraries.
3. Logging sensitive data. Adding authorization headers or PII as attributes triggers PII filters and may violate compliance. Strip secrets before calling the API.
4. Attaching on every span. Some teams call addCustomAttribute inside each database/HTTP wrapper. This duplicates the attribute on every span, inflating payload size and creating noise. Set once per transaction.

Verification Checklist

After deploying attribute instrumentation, confirm it works end-to-end:

1. APM UI Transaction Trace

Open a recent transaction → Attributes tab. Verify your custom key (e.g., tenantId) appears with the expected value. If missing, the call happened after transaction end or the agent didn't initialize.

2. NRQL Presence and Cardinality

SELECT count(*) FROM Transaction 
WHERE tenantId IS NOT NULL 
FACET tenantId LIMIT 100

Run in the New Relic Query Builder. Confirms the attribute is ingested and shows cardinality. If count is zero, check agent logs for initialization errors.

3. Agent Startup Logs

Tail the agent log (stdout or /var/log/newrelic/) on startup. Look for: - INFO: New Relic agent started - Absence of WARN: Attribute 'tenantId' exceeds size limit or WARN: Attribute filtered by PII rules

4. Distributed Trace Propagation

In the distributed trace waterfall view, click a downstream span → Attributes. The same tenantId should appear. If not, verify distributed_tracing.enabled and that the upstream service uses a supported protocol (HTTP, gRPC, Kafka, JMS).

Version Assumptions and Upgrade Notes

• Node.js agent ≥ 8.0: addCustomAttribute is synchronous and context-aware. Earlier versions used addCustomAttributes (plural) with different semantics.
• Java agent ≥ 6.0: addCustomParameter works in async contexts (CompletableFuture, Reactor). Requires class_transformer for automatic instrumentation.
• Python agent ≥ 5.0: add_custom_attribute supports asyncio via contextvars. WSGI middleware works automatically; ASGI needs newrelic.agent.initialize() before app creation.
• Go agent ≥ 3.10: Uses nrtxn.FromContext; requires passing context.Context through call chains.

Agent configuration (newrelic.yml, newrelic.js, environment variables) can disable attribute collection entirely (attributes.enabled: false) or exclude specific keys. Audit config if attributes disappear after an upgrade.

Practical Rollout Strategy

1. Start with one high-value, low-cardinality attribute (tenantId or environment) in a single service.
2. Verify in APM UI and NRQL for 24 hours.
3. Add the same attribute to upstream/downstream services using shared middleware library.
4. Build a dashboard faceting latency/error rate by the new attribute.
5. Only then consider additional attributes (feature flag, plan tier) after cardinality review.

This incremental approach avoids surprise ingest spikes and lets you validate propagation before expanding scope.