Event MPM thread pool limits and keep-alive latency under concurrent load

Identifying the Bottleneck

To distinguish between thread-pool queuing and keep-alive hold time, monitor the mod_status page (server-status) and the OS-level TCP stack. The key difference lies in whether the connection is actively occupying a worker thread or merely occupying a socket.

Observable Metrics

• Thread-Pool Queuing: This is indicated when BusyWorkers reaches the MaxRequestWorkers limit. At this point, new connections cannot be accepted by Apache and accumulate in the operating system's TCP listen queue. You can verify this using ss -ln or netstat -ln; a growing Recv-Q on the HTTP port confirms that the application is unable to accept connections fast enough.
• Keep-Alive Hold Time: In the Event MPM, keep-alive connections are handed off to a dedicated listener thread and do not occupy a worker thread until a new request arrives. Therefore, a high number of connections in Keep-Alive state (visible in mod_status) does not directly cause thread exhaustion, but it does consume file descriptors and memory.

Configuration Evaluation Order

When tail latency appears only under concurrency, evaluate MaxRequestWorkers first. This is the hard ceiling for concurrent request processing. If this limit is hit, requests queue at the OS level, creating the latency spikes described.

Only after ensuring MaxRequestWorkers is sufficient for your peak concurrent request volume should you tune KeepAliveTimeout. Reducing this timeout is a secondary optimization to reclaim memory and file descriptors, not a primary fix for thread-pool queuing.

Safe Assessment Strategy

To assess the interaction between these settings without risking production stability, use a staged load test with the following steps:

1. Baseline: Run a load test with a low concurrency level to establish a latency baseline.
2. Stress Test: Gradually increase concurrency until BusyWorkers hits MaxRequestWorkers. Observe the point where tail latency spikes and check the OS Recv-Q.
3. Resource Check: Before increasing MaxRequestWorkers, verify the OS file descriptor limit using ulimit -n. The limit must be higher than MaxRequestWorkers plus other system overhead to avoid "Too many open files" errors.
4. Timeout Variation: With a stable MaxRequestWorkers, vary KeepAliveTimeout (e.g., from 5s down to 2s) and monitor the rate of new TCP handshakes via netstat to ensure connection churn does not introduce new latency.

Missing Diagnostic Detail: To refine this recommendation, please provide the current ulimit -n value and the average memory footprint per Apache child process.