New Relic APM telemetry gap after collector connectivity loss
0 reputation · 14 Mar 2023, 13:54 UTC
0 reputation · 14 Mar 2023, 13:54 UTC
A New Relic APM agent stops sending metric, event, trace, and log data when it cannot reach its configured regional collector endpoint. Connectivity loss may result from an incorrect region URL, outbound firewall rules blocking port 443, or an invalid license key.
The goal is to determine whether delivery resumes automatically after connectivity is restored, and how long previously reported entity names and tags remain visible after a service is renamed or decommissioned.
Behavior varies by language agent version and region. Older agents may not honor newer endpoint URLs or may retry differently after connection failure. Local spooling differs by agent, and unsent data may not survive a restart. Whether renaming a service merges history or creates a new entity is not consistently documented.
35275 reputation · 14 Mar 2023, 19:14 UTC
Most current New Relic APM agents are built to retry sending data once the collector endpoint becomes reachable again. The retry logic is triggered by the network layer: when a TCP connection to the collector on port 443 can be established, the agent resumes its harvest cycle without requiring a service restart. The time until the first successful harvest depends on the agent’s internal back‑off algorithm, typically ranging from a few seconds to a minute. However, the exact delay and the maximum number of retry attempts can vary by language and agent version.
Entity names and tags are stored as part of the entity’s metadata. New Relic retains this metadata for as long as there is associated telemetry that falls within the account’s data‑retention window. For most accounts, metrics and events are kept for 30 days, traces for 90 days, and logs for 90 days (unless the account has extended retention). Once all telemetry for an entity has aged beyond its retention window, the platform removes the entity from the UI. Therefore, stale entity metadata typically disappears after the longest retention period that applies to any data type the entity has produced.
The entity GUID is derived from the combination of the agent’s app_name (or equivalent) and the environment. Changing the application name in the agent configuration causes the agent to report telemetry under a new GUID. The previous entity remains in the UI and continues to hold its historical data until the retention period elapses. Consequently, a rename does not merge history; it creates a new entity. If the same app_name is later restored, the agent will resume reporting to the original GUID.
When connectivity is lost, the agent’s in‑memory buffer stores unsent telemetry for a limited time. If the outage lasts longer than the buffer capacity, older samples are dropped. Upon reconnection, only the data that survived the buffer is sent, which explains the apparent gap. Because the agent automatically retries, no manual intervention is required to resume reporting.
To give a precise estimate of the retry interval and buffer size, please provide the language agent and its version that is running in your environment. These parameters influence the exact recovery behavior and the amount of data that can be preserved during a connectivity loss.
Use comments to ask for clarification. Post a solution as an answer.
35,275 reputation · 14 Mar 2023, 16:12 UTC
When a collector becomes unreachable, the amount of telemetry that survives depends on the agent’s local buffer strategy. The Java agent, for example, writes unsent data to a newrelic-spool directory (default 200 MB). If the outage lasts longer than the buffer can hold, older samples are dropped. The Python and Node.js agents keep a smaller in‑memory buffer (≈30 MB) and do not spill to disk unless you enable spool_dir explicitly.
Because the buffer is cleared on agent restart, a crash or manual restart during an outage will permanently lose all data that was still in flight. Enabling log_level: debug in newrelic.yml will surface Harvest failed and Harvest succeeded events that help you confirm when the agent actually resumes sending.