Fix 'Too many open files' in systemd with a LimitNOFILE Drop-In

The problem and the fix

When a daemon under systemd fails with "Too many open files", the cause is usually the inherited file-descriptor limit, not the application itself. The practical fix is to create a drop-in override that sets LimitNOFILE= under the unit's [Service] section, run systemctl daemon-reload, and restart the service.

How systemd applies the limit

Before systemd execs a service process, it calls setrlimit(RLIMIT_NOFILE, ...) with the value from the unit configuration. The LimitNOFILE= directive accepts either a single integer, which sets both the soft and hard limits to the same value, or a soft:hard pair. The pair syntax is not available in older systemd releases, so check your version if you need asymmetric limits.

A soft limit is the value the kernel enforces for new descriptors; a process can raise it up to the hard limit. A hard limit is the ceiling a process cannot exceed without CAP_SYS_RESOURCE. Setting a single integer is usually sufficient for services that need many concurrent connections or open files.

Worked example: raising nginx's limit

Run the following as root or with sudo. Replace nginx with your unit name.

sudo systemctl edit nginx

This opens an editor and creates a drop-in file such as /etc/systemd/system/nginx.service.d/override.conf. Add:

[Service]
LimitNOFILE=65535

Save and exit. Then reload the manager and restart the unit:

sudo systemctl daemon-reload
sudo systemctl restart nginx

Before restarting, confirm the parsed value:

systemctl show nginx -p LimitNOFILE

Expected check: the output should show LimitNOFILE=65535 or the pair you configured. After the restart, verify the kernel applied it to the running process:

cat /proc/$(pidof nginx)/limits

Look for the Max open files row. The soft and hard columns should reflect the value you set. If the process forks workers, check the master PID or the specific worker PID you care about.

Kernel-wide ceilings

The per-process limit cannot exceed the kernel's fs.nr_open sysctl, and the total number of open files across all processes cannot exceed fs.file-max. If either ceiling is lower than your requested value, the kernel silently clamps the limit. Check both:

sysctl fs.nr_open fs.file-max

If fs.nr_open is below your target, create a file under /etc/sysctl.d/, for example /etc/sysctl.d/99-fs-limits.conf, containing:

fs.nr_open = 65535

Apply it with sudo sysctl --system or reboot. Raising fs.file-max follows the same pattern, but only do so if you expect the aggregate load to justify it.

Common mistakes

• Using ulimit -n in an ExecStart wrapper. Changing the limit in a shell before launching the daemon is unreliable because the shell may exec the final binary in a way that discards the setting, and systemd's own limit handling can overwrite it on restart. Use LimitNOFILE= instead.
• Editing the main unit file directly. Package updates can overwrite files under /usr/lib/systemd/system/ or /etc/systemd/system/. A drop-in under nginx.service.d/ survives updates and is easier to audit.
• Forgetting systemctl daemon-reload. The running systemd manager does not re-read unit files until you reload. Without it, a restart uses the old configuration.
• Ignoring inherited defaults. If /etc/systemd/system.conf sets DefaultLimitNOFILE=, a unit without an explicit LimitNOFILE= inherits that global value, which may be lower or higher than the traditional 1024.

Limits and risks

Every open file descriptor consumes kernel memory. Setting a very high limit does not allocate that memory up front, but it removes a safety rail: a leaking or runaway process can exhaust system memory and trigger the OOM killer. Choose a value based on measured need, not a round number.

If the service is socket-activated, runs under a different user, or is started by another parent process, verify that no other limit source overrides the systemd setting. PAM's pam_limits.so can apply limits for login sessions, but it does not govern systemd services started directly by the manager. For socket-activated units, the limit on the listening socket is inherited from the socket unit, not the service unit.

Verification checklist

1. Run systemd-analyze verify nginx to catch syntax errors in the drop-in before reloading.
2. Run systemctl show nginx -p LimitNOFILE to confirm the manager parsed the value.
3. After restart, inspect /proc/<pid>/limits for the Max open files row.
4. Run sysctl fs.nr_open fs.file-max to ensure the kernel ceilings are not clamping your target.

Rolling back

Because this change modifies systemd configuration, rollback is straightforward. Remove the drop-in file:

sudo rm /etc/systemd/system/nginx.service.d/override.conf
sudo systemctl daemon-reload
sudo systemctl restart nginx

The service returns to the inherited default. If you raised fs.nr_open or fs.file-max, remove or edit the corresponding file under /etc/sysctl.d/ and run sudo sysctl --system.