DigitalOcean Load Balancer to Droplets: which boundary to scale when latency appears only under concurrent load

Question

We run an HTTP service behind a DigitalOcean Load Balancer forwarding to a small pool of Droplets. Under low traffic, p99 latency is flat. When we ramp concurrent clients, tail latency climbs sharply, and we cannot tell whether the bottleneck is the load balancer tier or the backends.

Our understanding of the documented behavior: the load balancer terminates client connections and opens separate connections to the Droplets, so tail latency depends on both sides. Health checks that turn slow under load can flap backends in and out of rotation, and connection reuse between the load balancer and Droplets can force fresh handshakes during bursts. On the Droplet side, CPU steal, conntrack limits, or application thread pools only saturate at concurrency, which looks identical to network latency without metrics.

The unresolved decision is the scaling axis: resize the load balancer to a larger tier versus adding more backend Droplets. DigitalOcean documents per-tier connection and request limits, but we have not found guidance on how to attribute queuing to one boundary or the other before spending on either.

Specifically:

Which load balancer and Droplet metrics best distinguish load balancer queuing from backend saturation during a concurrency ramp?
Is there a documented way to observe connection churn or health-check-driven backend removal as a latency cause?
Given a fixed budget, what decision rule do operators use to choose between a larger load balancer tier and more Droplets?

Assume current-generation load balancer tiers; exact limits need verification against current docs.

Tasadduq Burney · Answer

Answer the core question first
To decide whether to resize the load balancer (LB) tier or add more Droplets, compare two sets of metrics taken at the same time during a concurrency ramp:

LB‑side metrics – concurrent connections, new connections per second, requests per second, and the backend queue length reported by the LB dashboard.
Droplet‑side metrics – CPU usage (including steal time), run‑queue length, web‑server worker pool occupancy, conntrack table usage, and the count of inbound TCP/TLS handshakes per second.

If the LB metrics hit their tier limits while Droplet metrics remain well below saturation, the tail latency originates at the LB. If the Droplet metrics saturate first and the LB shows plenty of headroom, the bottleneck is on the backends.
Confirming the source with a controlled experiment
Run the same concurrency ramp from an internal client directly against a single Droplet (bypassing the LB). If tail latency disappears, the LB tier or the LB‑to‑Droplet path is the culprit; if it persists, the backend is the bottleneck.
Practical decision rule (budget‑aware)

LB‑bound – concurrent connections or new connections per second reach the documented tier ceiling, and Droplet metrics (CPU, worker pool, run queue) stay under 70 %. Scale to the next LB tier.
Backend‑bound – Droplet CPU > 70 % or worker pool full while LB metrics show 10 % usage of the tier limits. Add more Droplets (or increase the pool size).
Both idle – neither tier nor backends hit limits. Look for health‑check flapping or excessive connection churn (high inbound handshake rate). Adjust health‑check thresholds or enable connection reuse on the LB.

Key diagnostics to collect now

Current LB tier limits: maximum concurrent connections, requests per second, and new connections per second.
Health‑check configuration: interval, threshold, and whether it uses the same port as real traffic.
Connection‑reuse settings on the LB (keep‑alive, TLS session caching).

Next step – ask for missing detail
To refine the recommendation, could you share the health‑check interval and threshold values you’re using, and whether the health‑check shares the same port as your application traffic? This will help determine if health‑check flapping is inflating tail latency.

DigitalOcean Load Balancer to Droplets: which boundary to scale when latency appears only under concurrent load

1 answer

Answer the core question first

Confirming the source with a controlled experiment

Practical decision rule (budget‑aware)

Key diagnostics to collect now

Next step – ask for missing detail

Your perspective belongs here.

0 question comments