New Relic APM: Stale Transaction Trace Data Persists After Deletion
0 reputation · 19 Jul 2021, 15:40 UTC
0 reputation · 19 Jul 2021, 15:40 UTC
New Relic APM stores transaction trace data in an in‑memory cache to speed up UI rendering. The eviction policy is time‑based, yet traces can remain cached beyond their expected window when certain load patterns occur. When a trace is removed from the database—either by retention policy or manual deletion—the cached copy may still appear in the UI, presenting stale results.
The cache key incorporates only the transaction name and timestamp, omitting the full trace ID. This can lead to key collisions when multiple traces share the same name and timestamp but differ in ID, causing the UI to display an outdated trace.
New Relic documentation notes a pending fix but offers no release date, and there is currently no exposed API to force cache invalidation.
31775 reputation · 20 Jul 2021, 00:46 UTC
Short answer: the stale trace you're seeing is almost certainly coming from the UI-side cache, not the database. The deletion worked; the cache just hasn't caught up. There is no user-facing way to flush a single trace from that cache, and the cache key design is not something you can change yourself.
The likely explanation is a sliding (access-refreshed) expiration rather than a fixed TTL. With sliding expiration, every time someone opens the trace in the UI, the cache entry's timer resets. A trace that gets viewed repeatedly—say, one linked in an incident channel or a dashboard—can stay cached long past its nominal lifetime, even after the underlying record is removed by retention or manual deletion. Under low traffic the same cache would appear to evict correctly, which makes the behavior look inconsistent.
This is the likely explanation, not a confirmed one. What is confirmed is the second problem: the cache key is built from transaction name plus timestamp only, without the trace ID. Two traces with the same name and timestamp collide on one key, so the UI can serve an older trace's payload for a newer request. That collision can make staleness look permanent even when eviction is working.
No—not from your side. The key design lives in New Relic's product code, so including the trace ID would require a product release. If it were changed, the trade-offs would be predictable: key cardinality goes up, collision-driven staleness largely disappears, but cache hit ratio drops and memory usage grows because near-identical entries no longer share a slot. The eviction algorithm itself would stay time-based, so sliding-TTL staleness could still occur. Given that New Relic has acknowledged a pending fix with no date, treat this as their problem to ship, not something to engineer around in your account.
There is no API, NerdGraph mutation, or UI control to invalidate a specific trace or the whole trace cache. In practice the cache clears when:
One detail that would sharpen this: the configured TTL for the trace cache in your account. If observed persistence exceeds that TTL under zero access, escalation to support is the right move rather than further local debugging.
Use comments to ask for clarification. Post a solution as an answer.
No question comments on this page.