Diagnosing whether systemd LimitNOFILE or TasksMax caps cause connection pool exhaustion on Rocky Linux 8/9

Answer – Quick Decision Path

1. Identify the error type that appears when the connection pool stalls:

• EMFILE (Too many open files) ⇒ LimitNOFILE is the likely ceiling.
• Fork/clone failures such as cannot create new native thread or clone(2): Resource temporarily unavailable ⇒ TasksMax is the likely ceiling.

2. Verify live consumption against the configured ceilings using the commands below (they work on both cgroup v1 and v2).

Live FD and Task Metrics

1. systemctl show <unit> -p LimitNOFILE,TasksMax,TasksCurrent,MainPID
   Shows the configured limits and the unit’s main PID.
2. ls /proc/$(systemctl show -p MainPID --value <unit>)/fd | wc -l
   Counts open file descriptors for the unit’s process.
3. cat /proc/$(systemctl show -p MainPID --value <unit>)/status | grep Threads
   Shows the current thread count.
4. cat /sys/fs/cgroup/<slice>/<unit>/pids.current (cgroup v1)
   or cat /sys/fs/cgroup/system.slice/<unit>/pids.current (cgroup v2)
   Gives the live task count enforced by the cgroup.

3. Compare the live counts to the ceilings. If Threads or pids.current is close to TasksMax, the task ceiling is hit. If the FD count is near LimitNOFILE, the file‑descriptor ceiling is hit.

Deciding Between Raising Limits or Shrinking the Pool

• Raise the ceiling if the application is correctly sized for the workload and the live count is just below the ceiling. A drop‑in override (systemctl edit <unit>) is safe and keeps the service logic unchanged.
• Reduce the pool if the live count consistently exceeds the ceiling or if the ceiling is set far above the application’s design. Scaling the pool down aligns the service with the system’s limits and avoids waste.
• Always re‑test under load after either change to confirm the bottleneck is resolved.

Kernel‑Wide Limits to Verify First

1. sysctl fs.file-max – maximum number of file descriptors system‑wide.
2. sysctl fs.file-nr – current usage (allocations, inodes, etc.).
3. sysctl kernel.pid_max – maximum PID value; TasksMax defaults to roughly 15% of this.
4. sysctl kernel.threads-max – maximum number of threads system‑wide.
5. Check /etc/security/limits.d for per‑user nofile and nproc limits that might be hit before systemd.

Raising a service’s LimitNOFILE or TasksMax above these kernel caps will have no effect; the kernel limits win silently.

What to Ask If You’re Uncertain

Do you see EMFILE errors in the application logs, or do you see fork/clone failures such as cannot create new native thread?

Summary

Use the commands above to map live usage to configured ceilings, verify kernel limits, and then either raise the appropriate systemd ceiling or shrink the application pool. This approach ensures you hit the real bottleneck, not an arbitrary cap.