Diagnosing DigitalOcean Load Balancer Health Check Failures When Droplets Respond Directly

Recognizable Condition

The DigitalOcean Load Balancer UI marks one or more backend Droplets as down, yet a curl from another Droplet in the same VPC returns 200 OK on the same endpoint. Traffic is therefore not routed to those Droplets, causing intermittent 502/503 errors for clients.

Cause / Diagnostic Quick‑Reference Table

Symptom	Likely Cause	Quick Test
Health check path returns 404	Path/port mismatch (e.g., LB checks / but app serves /health)	curl -I http://:/health
Intermittent down after deploy	Startup race – app not ready before LB deregistration grace (30 s) expires	Check app logs for “listening on …” timestamp vs LB event log
All Droplets unhealthy, but direct curl works	Firewall blocking LB health‑check CIDR 198.18.235.0/24	nc -zv from a host in that CIDR (or use doctl compute firewall to list rules)
HTTPS health check fails with cert error	Backend expects TLS but LB health check configured as HTTP, or self‑signed cert without validation disabled	curl -k https://:/health
Flapping (healthy ↔ unhealthy every few seconds)	Interval/timeout/threshold too aggressive	Review LB health‑check settings: interval, timeout, unhealthy threshold

Ordered Diagnostic Checks

1. Verify health‑check path and port – In the LB settings confirm Health Check Path matches the exact endpoint your application exposes (including leading slash). Ensure the port matches the listener port on the Droplet (often 80, 443, or a custom app port).
2. Check timeout vs application latency – Default timeout is 5 s. If your /health endpoint does DB checks or external calls, raise timeout to 10‑15 s. In the UI or via doctl:

   doctl compute load-balancer update  \
         --health-check-timeout 15 \
         --health-check-interval 10 \
         --health-check-unhealthy-threshold 3

   Run locally with a doctl‑authenticated session (API token with read/write on Load Balancers).
3. Confirm firewall allows LB health‑check CIDR – The LB health checks originate from 198.18.235.0/24, not from the LB’s public IP. Add an inbound rule on the Droplet’s Cloud Firewall (or host‑level ufw/iptables) for TCP from that CIDR.

   doctl compute firewall create \
     --name allow-lb-health \
     --inbound-rules "protocol:tcp,ports:,addresses:198.18.235.0/24" \
     --droplet-ids 

   Test with a single Droplet first; Cloud Firewalls apply to all tagged Droplets.
4. Match TLS configuration – If the backend serves HTTPS, set the LB health‑check protocol to HTTPS. For self‑signed certs you must disable certificate validation via the API (UI does not expose this):

   doctl compute load-balancer update  \
     --health-check-protocol https \
     --health-check-port 443 \
     --health-check-tls-skip-verify true

   Requires API token with load_balancer write scope.
5. Eliminate startup race – Ensure the application binds to the health‑check port and returns 200 before the LB’s 30 s deregistration grace period ends. Add a readiness probe in your container/process manager that only signals ready after the health endpoint is live.

Fixes Tied to Findings

• Path/port mismatch – Update LB health‑check path/port to match the actual endpoint (e.g., /health on port 8080).
• Timeout too low – Increase health_check_timeout as shown in step 2.
• Firewall blocking – Apply the inbound rule for 198.18.235.0/24 (step 3). Verify with doctl compute firewall get that the rule appears.
• TLS mismatch – Switch protocol to HTTPS and, if needed, enable tls_skip_verify (step 4).
• Flapping – Adopt minimum stable values: interval ≥ 10 s, timeout ≥ 10 s, unhealthy threshold ≥ 3.

Verification After Changes

1. Watch the LB UI – the affected Droplets should show healthy within 2‑3 health‑check intervals (≈20‑30 s with the recommended settings).
2. Run doctl compute load-balancer get and confirm every instance in droplet_ids reports status: healthy.
3. Generate synthetic traffic (e.g., hey -n 1000 -c 10 https:///) and verify no 502/503 responses; inspect LB access logs for even distribution across Droplets.
4. Simulate a failure: stop the app on one Droplet (systemctl stop myapp) and confirm the LB marks it unhealthy within the configured threshold. Restart the app and verify automatic recovery without manual intervention.

Escalation Criteria

If after completing all five checks the Droplets remain down:

• Collect LB event logs (doctl compute load-balancer get --format Events) and Droplet syslog for the exact health‑check timestamps.
• Open a DigitalOcean support ticket with the LB ID, affected Droplet IDs, and the full output of the verification steps.
• Consider temporary removal of the Droplet from the LB pool (via UI or doctl compute load-balancer remove-droplets) to isolate whether the issue is LB‑side or Droplet‑side.

Limitations & Practical Check

This guide covers the most common misconfigurations. It does not address network‑level issues outside DigitalOcean’s control (e.g., upstream ISP filtering) or application‑level bugs that cause the health endpoint to return non‑2xx intermittently. The definitive practical check is the doctl compute load-balancer get command showing all backends as healthy followed by a load test with zero 5xx responses.