Select() latency spikes under high concurrent file descriptor load

Select() Latency Under Heavy FD Load

The POSIX select() API is widely used for monitoring multiple file descriptors, but its performance under concurrent high‑load conditions is uncertain. The specification intentionally leaves the algorithm for evaluating readiness unspecified, allowing implementations to choose from linear scans, bitmap checks, or kernel‑level event notification. In practice, many modern Unix kernels iterate over the entire set of descriptors for each call, which can lead to O(n) latency spikes when hundreds or thousands of descriptors are monitored concurrently.

We need to understand how a given kernel behaves when select() is invoked on a large set of descriptors while many concurrent I/O operations are in flight. The goal is to determine whether the observed latency is due to the unspecified algorithm choice and how it scales with the number of descriptors.

Key constraints and uncertainties:

• Different kernel releases or configurations may use distinct strategies for select().
• There is no documented guarantee of latency characteristics in the POSIX standard.
• Performance may only become apparent under concurrent, high‑load scenarios.

Specific questions for investigation:

1. What algorithm does the current Unix kernel employ for select() when monitoring thousands of file descriptors?
2. How does the per‑call latency of select() scale with the number of monitored descriptors under concurrent I/O activity?
3. Which kernel configuration options or patches influence the performance characteristics of select() in high‑concurrency environments?