Diagnosing Intermittent Agent Lost Errors in Azure DevOps Self‑Hosted Agents

Direct answer

Agent Lost is a server-side state: Azure DevOps marks a self-hosted agent lost when its long-polling HTTPS listener stops sending heartbeats within the server's absence window. Under high concurrency, the realistic causes are ephemeral TCP port exhaustion on the agent host or an intermediary firewall/proxy dropping the idle long-poll connection. Increasing the maximum concurrent jobs does not widen the heartbeat window and will probably worsen the symptom by adding outbound connections. No supported public agent configuration flag sets the heartbeat timeout; that timing is internal and agent-version dependent.

Confirmed facts vs. likely explanation

Confirmed

The agent writes heartbeat send/receive and connection-error entries to its _diag folder; the server transitions the agent to Lost when those check-ins stop. There is no documented public setting for heartbeat interval or timeout.

Likely

During parallel jobs the agent opens more outbound sockets. If the ephemeral port pool is already saturated—often visible as high TIME_WAIT counts—new listener connections fail. Alternatively, a stateful firewall or proxy with an aggressive idle TCP timeout can silently tear down the long-poll, producing the same missed-heartbeat signature.

Steps to confirm TCP port exhaustion

1. Match each Agent Lost event to the exact _diag timestamp. If the log shows connection errors or missing sends at that moment, the listener is failing locally or at the network edge.
2. Capture port state during a high-concurrency run:

   # Windows
   netstat -an | findstr /R "ESTABLISHED TIME_WAIT"
   netsh int ipv4 show dynamicport tcp
   # then check Event Viewer for TCP exhaustion (event ID 4227)
   
   # Linux
   ss -tan state time-wait | wc -l
   cat /proc/sys/net/ipv4/ip_local_port_range
   dmesg | grep -i "port allocation"

3. If counts approach the dynamic range or allocation errors appear, ephemeral port exhaustion is implicated.
4. Correlate the same window with proxy/firewall logs. An idle-timeout disconnect there points to network policy rather than local limits.

Does raising concurrent jobs help?

No. Maximum concurrent jobs controls parallel job execution, not heartbeat frequency. More jobs mean more simultaneous outbound connections, which can accelerate port-pool pressure and increase Agent Lost frequency. Use it only after CPU, memory, and port headroom are confirmed.

Heartbeat timeout configuration

Do not add undocumented keys to .agent or runner configuration files to tune heartbeat timeouts. Behavior is internal to the agent package and server; inventing flags is unsupported. Record the installed agent version and platform, then compare observed _diag intervals against the expected check-in window for that release.

Quick isolation test

Temporarily set the agent to a single concurrent job and reproduce the load. If Agent Lost stops, local resource pressure is the likely cause. If it persists, focus on proxy/firewall idle timeouts and TCP keepalive behavior between the agent host and Azure DevOps.