Diagnosing File Descriptor Exhaustion: EMFILE and ENFILE on Unix

The recognizable condition

A long-running daemon or network service suddenly rejects new work. accept(), open(), socket(), or pipe() calls fail, and the application logs "Too many open files" or a similar resource-exhaustion message. On Unix, this is almost always file-descriptor exhaustion. The kernel enforces two separate ceilings: a per-process limit (RLIMIT_NOFILE) and a system-wide limit. When the process ceiling is hit, the call fails with EMFILE. When the system ceiling is hit, it fails with ENFILE. Application logs rarely distinguish the two, so the first task is to tell them apart.

Do not confuse these errors with EACCES (permission denied) or ENOENT (no such file or directory). Those indicate a path or permission problem, not exhaustion, and raising limits will not help.

Cause and diagnostic mapping

Observed symptom	Likely errno	Scope	First diagnostic
One service fails; others on the same host work	EMFILE	Per-process	Compare /proc/<pid>/fd count to ulimit -n
Multiple unrelated services fail simultaneously	ENFILE	System-wide	Compare /proc/sys/fs/file-nr to /proc/sys/fs/file-max (Linux)
"Permission denied" on open()	EACCES	Not exhaustion	Check file ownership and mode bits
"No such file or directory" on open()	ENOENT	Not exhaustion	Check path spelling and mount points

Ordered checks

1. Capture the exact errno

Check the application log or use strace to see which syscall fails and with which errno. If the log only says "Too many open files," treat it as EMFILE or ENFILE and proceed.

2. Compare process descriptor count to its limit

On Linux, run as root or the same user that owns the process:

ls /proc/<pid>/fd | wc -l
ulimit -n
cat /proc/<pid>/limits | grep "open files"

If the descriptor count equals the soft limit from ulimit -n (or the "Soft Limit" in /proc/<pid>/limits), the process is hitting EMFILE. On BSD systems, use fstat -p <pid> or lsof -p <pid> and compare the line count to the output of ulimit -n.

3. Check system-wide file usage

If the process count is well below its limit, or if many processes are affected, check the kernel-wide counters:

cat /proc/sys/fs/file-max
cat /proc/sys/fs/file-nr

file-nr shows allocated, free, and maximum file handles. If the allocated value is at or near file-max, the system is hitting ENFILE. On BSD, check kern.maxfiles and kern.openfiles via sysctl.

4. Inspect descriptor types for leak patterns

List the descriptors and look for anomalies:

ls -l /proc/<pid>/fd

Look for large numbers of socket: entries in CLOSE_WAIT state (the peer closed but the application did not), many anonymous pipes, or regular files that should have been closed. A healthy process has a stable count; a leaking process shows a count that only grows over time.

Fixes tied to findings

If the process count equals its limit (EMFILE)

First, restart the process to clear leaked descriptors and restore service. This is a mitigation, not a fix. Then decide whether to raise the limit or repair the leak.

To raise RLIMIT_NOFILE for a new shell session, run ulimit -n 4096 before launching the daemon. Note that this only affects that shell and its children; it does not change running processes.

For a systemd service, add or edit the limit in the unit file:

[Service]
LimitNOFILE=4096

Then run systemctl daemon-reload and restart the service. The change applies only to the restarted process.

If system-wide usage is at the ceiling (ENFILE)

Raise the kernel maximum. On Linux:

sysctl -w fs.file-max=<new_value>

To persist across reboots, add fs.file-max=<new_value> to /etc/sysctl.conf or a file in /etc/sysctl.d/. Each open file consumes kernel memory, so set the value based on available RAM and expected workload, not arbitrarily. On BSD, adjust kern.maxfiles and kern.maxfilesperproc in /etc/sysctl.conf.

If descriptors are leaked by the application

Raising limits only delays the next failure. Identify the leak pattern from step 4 and fix the application code: close sockets after shutdown(), close files after use, and ensure error paths do not skip cleanup. If the leak is in a closed-source third-party binary, you cannot patch it directly; raising the limit and scheduling periodic restarts may be the only operational workaround.

Escalation criteria

• ENFILE persists after raising fs.file-max (or kern.maxfiles) and confirming adequate system memory.
• The leak is inside a closed-source third-party binary and the vendor cannot provide a fix.
• Production outages recur despite restarts and limit increases, indicating a deeper architectural or kernel issue.

Limitations and verification

Raising RLIMIT_NOFILE or fs.file-max masks leaks but does not repair them. Track descriptor counts over time with monitoring to confirm the root cause is resolved. Also remember that RLIMIT_NOFILE changes apply only to new sessions or restarted services; existing processes retain their previous limit.

To verify a fix, reproduce the original workload and confirm that open(), accept(), or socket() no longer return EMFILE or ENFILE. Then check that the descriptor count in /proc/<pid>/fd remains stable and well below the configured limit under sustained load.