Diagnosing and Fixing Data Ingestion Gaps in New Relic

The Problem: Missing Telemetry Buckets

Data ingestion gaps appear as empty intervals in your New Relic dashboards or NRQL (New Relic Query Language) charts. These gaps are often mistaken for application downtime, but they typically indicate a failure in the telemetry pipeline—the path between the agent on your host and the New Relic collector.

The primary goal is to determine if the data was never sent, was rejected by the collector, or was dropped due to host-level resource exhaustion.

Quick Diagnostic Reference

Symptom	Likely Cause	Primary Diagnostic Tool
Intermittent gaps across all metrics	Network egress or Firewall instability	curl / Network logs
Total data silence for one host	Invalid License Key or Agent crash	Agent logs (stdout/stderr)
Data exists but under a different name	Environment variable override	NRQL names()` query
Gaps coinciding with CPU spikes	Buffer overflow/Resource contention	Host OS metrics

Step-by-Step Investigation

1. Confirm the Gap via NRQL

Before investigating the infrastructure, verify that the gap is a data ingestion issue and not a visualization error. Run a query to count events in small time buckets.

SELECT count(*) FROM Transaction FACET bucket(1 minute) SINCE 1 hour ago

If the result shows zero counts for specific minutes while the application was known to be running, you have a confirmed ingestion gap.

2. Verify Network Reachability

Agents communicate with New Relic collectors over port 443. Firewalls or proxy settings can intermittently drop these packets.

Run this command from the affected host (replace YOUR_LICENSE_KEY with your actual key) to test the ingest API independently of the agent:

curl -v -X POST -H "X-Insert-1": "YOUR_LICENSE_KEY" -d "metric=test_connectivity,value=1" https://insights-collector.newrelic.com/v1/accounts

Expected Result: An HTTP 200 OK response. If you receive a 403, your license key is invalid. If the request hangs, port 443 is blocked or the endpoint is not whitelisted.

3. Inspect Agent Logs for Handshake Failures

Check the logs for the New Relic Infrastructure agent or APM agent. Look specifically for these indicators:

• HTTP 403: Indicates a license key mismatch. This often happens during automated deployments where environment variables are not passed correctly.
• HTTP 408: Indicates a request timeout, suggesting the agent cannot reach the collector within the allotted window.
• Heartbeat Failures: In Infrastructure agent logs, look for "heartbeat" errors, which signal a breakdown in the daemon-to-collector handshake.

4. Check for Resource Contention

When a host experiences extreme CPU or memory pressure, the agent may fail to flush its internal buffer to the network. This leads to "dropped packets" where telemetry is deleted locally to prevent the agent from crashing the host.

Compare the timestamps of your data gaps with the host's system logs (e.g., dmesg or /var/log/syslog) to see if OOM (Out of Memory) kills or CPU throttling occurred simultaneously.

5. Validate Application Naming

Sometimes data is being ingested, but it is routed to a different application name due to an override in the newrelic.js config or an environment variable like NEW_RELIC_APP_NAME.

Run this NRQL query to see if any "unexpected" application names appeared during the gap period:

SELECT uniqueCount(appName) FROM Transaction SINCE 1 hour ago

Resolution Matrix

• If Network Test Fails: Update firewall rules to allow egress traffic to New Relic collector endpoints on port 443.
• If 403 Errors Found: Verify the license key in the agent configuration file or secrets manager.
• If Resource Contention Found: Increase host resources or tune the agent's reporting interval to reduce the frequency of bursts.
• If Naming Mismatch Found: Standardize the app_name across all deployment manifests.

Limitations and Verification

Warning: Avoid restarting the agent immediately upon discovering a gap. Restarting clears volatile log buffers and in-memory queues, which may erase the specific error codes needed to diagnose the root cause.

To verify the fix, monitor the Infrastructure tab in the New Relic UI. A healthy connection will show the host as "Healthy" consistently. If the host status toggles between "Healthy" and "Missing," the underlying network or resource issue persists.

Escalation Criteria

Escalate to New Relic Support if:

• The curl test to the ingest API fails with a 5xx server error.
• Agent logs show successful HTTP 200 responses, but data is still missing from the UI.
• Network connectivity is confirmed, but the agent fails to start despite correct permissions and license keys.