Diagnosing a Low Dynatrace Health Score Caused by Synthetic Monitoring Issues

Problem Statement

A Dynatrace service’s health score suddenly drops below the 80 % threshold. The UI shows a red trend line, but the root cause is unclear. The most common culprit in these scenarios is a synthetic monitor that is reporting high latency or error rates, or that has been inadvertently disabled.

Recognizable Condition

When the Health score widget shows a value < 80 % and the trend line is trending downwards, look for:

• Elevated error rate on the monitored API endpoint.
• High average latency reported by synthetic checks.
• Monitor status marked as Failed or Disabled.

Cause / Diagnostic Table

Potential Cause	Diagnostic Indicator
Elevated error rate on API endpoint	Service error rate chart spikes > 5 % during the issue window.
High synthetic latency	Latency chart > 2 s for the synthetic monitor.
Monitor disabled or failed	Synthetic monitor status shows Disabled/Failed.
Aggressive alert thresholds	Alert rule shows errorRate > 2 % or latency > 1 s.
Network connectivity problems	Ping or traceroute from Dynatrace server to target fails or times out.
Host resource contention	CPU > 90 % or high I/O wait during the issue window.

Ordered Checks

1. Refresh the Health Score Widget
   • Navigate to Service Overview > Health score and click Refresh.
   • Confirm the current score and trend line.
2. Verify Synthetic Monitor Status
   • Go to Synthetic Monitoring > Monitors.
   • Locate the monitor for the affected API endpoint.
   • Check the Status column for Failed or Disabled.
3. Inspect Error Rate & Latency Charts
   • Open the service’s Metrics tab.
   • Apply a time range that covers the health score drop.
   • Look for spikes in Error rate and Latency.
4. Check Alerting Rules
   • Navigate to Alerting > Rules.
   • Filter by the service or monitor name.
   • Verify threshold values and suppression settings.
5. Test Network Connectivity
   • SSH into the Dynatrace server or the environment where the monitor runs.
   • Run:

     ping -c 4 <target-host>
     traceroute <target-host>

   • Check for packet loss or high round‑trip times.
6. Inspect Host Resource Metrics
   • Open the host’s Metrics tab.
   • Look for CPU, memory, or I/O spikes during the same period.

Fixes Tied to Findings

• Elevated Error Rate
  • Investigate the API endpoint logs for exceptions.
  • Apply a quick patch or rollback the recent deployment if the error rate correlates with a code change.
• High Synthetic Latency
  • Check the backend service’s response time; if it’s legitimately high, adjust the synthetic monitor’s Threshold or interval.
  • Example: Update the monitor via API:

    curl -X PATCH \
      "https://{environment-id}.live.dynatrace.com/api/v1/synthetic/monitors/{monitor-id}" \
      -H "Authorization: Api-Token {api-token}" \
      -H "Content-Type: application/json" \
      -d '{"threshold": 2000}'

• Monitor Disabled or Failed
  • Re‑enable the monitor: Enable button in the UI or via API.
  • If the monitor fails repeatedly, review the test script for errors.
• Aggressive Alert Thresholds
  • Adjust the rule to a more realistic value, e.g., errorRate > 3 %.
  • Re‑apply the rule and observe the health score trend.
• Network Connectivity Issues
  • Check firewall or load‑balancer rules between Dynatrace and the target.
  • Validate that the Dynatrace server can resolve the DNS name.
• Host Resource Contention
  • Scale the host, add CPU/memory, or move the service to a less loaded instance.
  • Consider horizontal scaling or load‑balancing the API.

Escalation Criteria

1. If the health score remains < 80 % after the above fixes within 30 minutes, open a Dynatrace support ticket.
2. Include screenshots of the health score, synthetic monitor status, and relevant metrics.
3. If the issue is linked to a deployment, involve the release management team.
4. For persistent network problems, coordinate with the network operations team.

Limitations & Practical Verification

Dynatrace data is refreshed every 5 seconds for live metrics but may be cached in the UI. After making changes, wait at least 2 minutes before re‑checking the health score. Also, synthetic monitors run from Dynatrace’s data centers; if your endpoint is behind a VPN or requires authentication, ensure the monitor’s credentials are up to date.

Conclusion

By following the ordered checks, applying targeted fixes, and knowing when to elevate the issue, you can quickly restore a healthy service score in Dynatrace. Regularly review synthetic monitor thresholds and alert rules to prevent future score drops.