Using New Relic Custom Attributes to Isolate Java Latency Spikes

Problem: Latency spikes hide in plain sight

Your checkout API occasionally jumps to ~2 seconds response time, but CPU, memory, and throughput metrics look normal. Standard APM dashboards show no obvious outliers, making it hard to know whether the slowdown is tied to a particular business step, a specific user segment, or a downstream service call.

Thesis: Targeted custom attributes turn vague spikes into actionable clues

By attaching meaningful key‑value pairs to each New Relic trace, you can slice transaction data by the exact context that matters—such as a checkout step, promotion code, or tenant ID—and instantly see whether latency correlates with that context.

Section 1 – Instrumenting the Java code

The New Relic Java API lets you add attributes to the current traced method. Wrap the call in a try‑finally block so the attribute is removed even if an exception occurs.

import com.newrelic.api.agent.NewRelic;
import com.newrelic.api.agent.Trace;

@Trace(dispatcher = true)
public void processCheckout(String step) {
    // Add a custom attribute that describes the business step
    try {
        NewRelic.getAgent().getTracedMethod().addAttribute("checkoutStep", step);
        // … existing checkout logic …
    } finally {
        // Clean up to avoid leaking the attribute into unrelated traces
        NewRelic.getAgent().getTracedMethod().removeAttribute("checkoutStep");
    }
}

Where to run: Edit your Java source, rebuild, and redeploy the service. You need write access to the repository and permission to deploy a new version of the application.

Permissions: The Java agent must be installed and running with the default permissions; no extra New Relic role is required to add attributes.

Risks: Each attribute adds a few bytes to every trace. If you attach high‑cardinality values (e.g., raw user IDs), the attribute count can explode and breach your plan’s limits, increasing ingestion cost.

Section 2 – Querying the data with NRQL

Once the attribute is flowing, you can filter transactions directly in NRQL. The following query shows the average duration for the "payment” step over the last hour:

SELECT average(duration) 
FROM Transaction 
WHERE appName='CheckoutService' 
  AND checkoutStep='payment' 
SINCE 1 hour ago

Where to run: Open New Relic → Query Builder, paste the NRQL, and execute. No special permissions beyond standard query access are needed.

Expected check: The result table should return a numeric average duration. If you see a noticeable jump compared to the overall average, the attribute is highlighting a latency hotspot.

Worked example: Load test reveals the faulty step

In a simulated load test, the team instruments the processCheckout method as shown above, setting checkoutStep to values like "cart", "payment", "confirmation". After the test runs, they execute the NRQL query for each step.

• Overall average duration: 320 ms
• Average for checkoutStep='payment': 1 950 ms
• Other steps remain near the overall average.

The spike is isolated to the payment step, pointing the team to a discount‑calculation service that is called only during payment. They can now focus profiling or debugging on that service rather than guessing across the whole stack.

Limitations: The attribute adds overhead to each trace; if you later decide to instrument many steps or use high‑cardinality values (e.g., exact promotion codes), you may exceed the attribute limit defined in your New Relic plan. To mitigate, group low‑frequency values or hash identifiers before attaching them.

Actionable closing: Start small, iterate, and alert

1. Pick one high‑value flow (e.g., checkout) and add a single custom attribute that captures the most relevant business context.
2. Deploy, then verify in the UI under APM → Transactions → Trace details that the attribute appears with the expected value.
3. Run the NRQL query in Query Builder to confirm you can filter by the attribute and see latency differences.
4. Check Manage data → Custom attributes to ensure the attribute count stays within your plan’s limits.
5. If the attribute proves useful, consider adding more attributes, but keep an eye on cardinality.
6. Create an alert condition: When average(duration) for Transaction where checkoutStep='payment' > 1 second for 5 minutes, trigger a notification.

By following these steps, you turn opaque latency spikes into a clear, attribute‑driven signal that lets you act fast.