Systemd LimitNOFILE, Fix Open File Limits for Services

limits.conf isn't applied to a systemd service, so set LimitNOFILE in a drop-in, restart the unit and read the new limit back from /proc.

Published

Applies to systemd 240 and later, where services default to 1024:524288; OpenRC on Alpine; Docker 25 and containerd 2.0 and later

systemd LimitNOFILE sets how many files and sockets a service may hold open. You raised nofile in /etc/security/limits.conf, and ulimit -n in your SSH session shows the new number. The service still dies at peak with Too many open files. The service never read that file. Below, you set the limit where systemd reads it and prove the running process got it.

For the kernel side of a busy web server, such as the listen queue and port range, see Linux network tuning for web servers.

What breaks at peak

Every socket, pipe and open file takes one file descriptor. A proxy holds two per request, one to the client and one to the upstream. Traffic grows, the count climbs, and at the soft limit the next accept() or open() fails with errno 24, EMFILE. Each runtime prints it its own way:

accept4() failed (24: Too many open files)
OSError: [Errno 24] Too many open files
Error: EMFILE: too many open files, open '/app/views/index.html'
java.net.SocketException: Too many open files
http: Accept error: accept tcp [::]:8080: accept4: too many open files; retrying in 5ms

The first line is nginx, then Python, Node.js, Java and Go. The C library supplies the text for nginx and Python. On Alpine, musl words the same errno No file descriptors available, so nginx logs (24: No file descriptors available). Grep for (24: to catch both.

Clients get refused or reset connections, and the proxy in front returns 502s. The process stays up and its health check may still pass, so nothing restarts it.

limits.conf has no effect here. pam_limits reads /etc/security/limits.conf when a PAM session starts: an SSH login, su or sudo. Services start from PID 1 and never open a PAM session. The stock file from Linux-PAM says so in its header:

#This file sets the resource limits for the users logged in via PAM.
#It does not affect resource limits of the system services.

systemd gives each service DefaultLimitNOFILE, which has been 1024:524288 since systemd 240. That’s a soft limit of 1024 and a hard limit of 524288. A process may raise its own soft limit up to the hard limit without privileges. Node.js does that at startup, and so does Go since 1.19. Python, most C daemons and anything run through a shell wrapper keep 1024 until you change it.

The systemd LimitNOFILE fix

Find the limit the running process has, not the one your shell has:

systemctl show app -p LimitNOFILE -p LimitNOFILESoft
grep 'open files' /proc/$(systemctl show -p MainPID --value app)/limits
ls /proc/$(systemctl show -p MainPID --value app)/fd | wc -l

The first line shows what systemd will apply at the next start. /proc/<pid>/limits shows what the process has now, soft limit first. The last line counts descriptors in use. Run the last two against a worker as well as the main PID, since workers can set their own limit.

Set it in a drop-in. systemctl edit app opens a drop-in file and reloads systemd when you save:

# /etc/systemd/system/app.service.d/override.conf
[Service]
LimitNOFILE=65535
systemctl restart app

A single value sets the soft and hard limit to the same number. LimitNOFILE=65535:524288 sets them apart, which keeps the high hard limit for runtimes that raise their own soft limit. systemd applies limits when it starts the process, so systemctl reload doesn’t pick them up. Restart, then rerun the /proc check.

Size it. Take your peak concurrent connections, double it for a proxy, then add the files, log handles and listening sockets the process keeps open. Set the limit to about twice that. The ceiling is fs.nr_open, the most any one process may hold, and systemd raises it at boot on current releases. Check it with sysctl fs.nr_open.

I set a number and skip LimitNOFILE=infinity. infinity maps to fs.nr_open, often over a million. Some daemons loop over every possible descriptor at startup to close them, and a huge limit makes that loop slow.

Mind select(). The systemd docs warn that select() can’t handle descriptors above 1023. A program built on select() crashes or corrupts memory once it holds more than 1024. Most servers use epoll, but check older C programs and some language bindings before you raise their soft limit.

Use the app’s own setting where it has one. nginx workers take their limit from worker_rlimit_nofile in nginx.conf. The nginx high-traffic recipe sizes it against worker_connections. The unit’s LimitNOFILE still caps what the master may grant, so set both. Database servers have their own knobs too, so read the server’s docs before you change the unit.

Change the default for every service. A file in /etc/systemd/system.conf.d/ sets DefaultLimitNOFILE= for all units. It needs systemctl daemon-reexec and a restart of each service. I keep limits per unit instead, so a change to one service can’t reach the rest.

Containers. Docker 25 and containerd 2.0 dropped LimitNOFILE=infinity from their systemd units. Containers that used to inherit a huge limit can now start at the systemd default with a soft limit of 1024. Check from inside with ulimit -Sn and ulimit -Hn. Set it per container with docker run --ulimit nofile=65535:65535, or for all containers in /etc/docker/daemon.json:

{
  "default-ulimits": {
    "nofile": { "Name": "nofile", "Soft": 65535, "Hard": 65535 }
  }
}

Kubernetes has no pod field for nofile. Containers get the runtime’s limit, so on nodes you raise it with a LimitNOFILE drop-in on containerd.service and restart containerd. Do that in the node image and roll it out with new nodes.

Distro differences. The systemd releases below all carry the 1024:524288 default:

DistroService managerWhere the limit goesDefault for services
Ubuntu and Debiansystemd 255 on Ubuntu 24.04, 252 on Debian 12, 257 on Debian 13/etc/systemd/system/<unit>.service.d/override.conf1024 soft, 524288 hard
RHEL, Rocky, AlmaLinuxsystemd 252 on RHEL 9Same drop-in path1024 soft, 524288 hard
Amazon Linux 2023systemd 252Same drop-in path1024 soft, 524288 hard
AlpineOpenRCrc_ulimit="-n 65535" in /etc/conf.d/<service>The kernel default, 1024 soft and 4096 hard

Packaged units sometimes set their own LimitNOFILE, so a distro’s nginx or database unit can differ from the default. systemctl cat app shows the unit plus every drop-in, in the order systemd applies them. In Alpine containers there’s no OpenRC, so the container runtime’s limit applies.

Test it on your system

Use a staging host with the same unit file and drop-ins as production. Drive enough concurrent connections to reach your peak. A fixed arrival rate with slow responses holds connections open, so use k6’s constant-arrival-rate executor with preAllocatedVUs set high. For a raw connection count, wrk -c set to your peak with -d 10m works too.

Watch the descriptor count against the limit while it runs:

PID=$(systemctl show -p MainPID --value app)
watch -n 2 "ls /proc/$PID/fd | wc -l; grep 'open files' /proc/$PID/limits"
journalctl -u app -f | grep -E 'Too many open files|\(24: '

To prove your alerts fire, lower the limit on the running process in staging, then run the load again:

prlimit --pid "$PID" --nofile=2048:2048

prlimit changes a live process without a restart. Workers forked before the change keep their old limit, so run it on each worker PID too. Restart the service afterwards to get the unit’s value back.

Set your targets before the run. The descriptor count at peak should stay under half the limit. The logs should show no errno 24 lines, and the load test should show no connection resets. Under the lowered limit, the alerts below should fire before errors appear.

Keep it healthy

Three exporters cover this. Most Prometheus client libraries export process_open_fds and process_max_fds for their own process, where process_max_fds is the soft limit. For processes without a client library, such as nginx or a Python worker pool, run process-exporter. Its namedprocess_namegroup_worst_fd_ratio reports the fullest process in each group against its soft limit. node_exporter’s filefd collector exports the system-wide count as node_filefd_allocated and node_filefd_maximum.

groups:
  - name: linux-file-limits
    rules:
      - alert: ServiceNearOpenFileLimit
        expr: namedprocess_namegroup_worst_fd_ratio > 0.8
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "A {{ $labels.groupname }} process on {{ $labels.instance }} uses over 80% of its open-file limit"
          description: "Check grep 'open files' /proc/<pid>/limits. Raise LimitNOFILE in the unit's drop-in and restart, or find the leak if the count keeps climbing at steady load."
      - alert: ServiceAtDefaultSoftLimit
        expr: process_max_fds <= 1024
        for: 15m
        labels:
          severity: warning
        annotations:
          summary: "{{ $labels.job }} on {{ $labels.instance }} runs with a soft open-file limit of 1024 or less"
          description: "The service is on the systemd or kernel default. Add LimitNOFILE to its drop-in, or the container's nofile ulimit, before peak traffic reaches it."
      - alert: HostFileDescriptorsNearMax
        expr: node_filefd_allocated / node_filefd_maximum > 0.8
        for: 10m
        labels:
          severity: critical
        annotations:
          summary: "{{ $labels.instance }} has used over 80% of fs.file-max"
          description: "Every process on the host will soon fail to open files. Find the process holding the most with process-exporter or lsof, then raise fs.file-max."

ServiceAtDefaultSoftLimit catches the drift that causes most of these outages. A new image, a renamed unit or a lost drop-in puts the service back on 1024, and you find out at the next peak. I check it after every node image change.

Tune the 0.8 thresholds to how fast your count climbs. A service that goes from half to full in a minute needs a lower threshold and a shorter for.

Roll back

Save the current state before you change anything:

systemctl show app -p LimitNOFILE -p LimitNOFILESoft > /root/app-nofile-before.txt
systemctl cat app > /root/app-unit-before.txt

To roll back, remove the drop-in and restart:

rm /etc/systemd/system/app.service.d/override.conf
systemctl daemon-reload
systemctl restart app

On Alpine, remove the rc_ulimit line and run rc-service app restart. For Docker, remove the default-ulimits block, restart dockerd and recreate the containers.

A higher open-file limit seldom causes harm by itself. Roll back now if the service misbehaves after the change, which points to a select() program. Otherwise, revert off-peak, since the restart drops open connections.

More recipes

Further reading