Out of memory: Killed process in the kernel log means the host ran out of RAM and the kernel picked a process to end. At peak that process is often your app, your database or the agent that ships its logs. The fix is to decide which process goes first, before the kernel decides for you.
If the kill happened inside a Kubernetes pod, the pod’s memory limit set it off. Read OOMKilled and exit code 137 for that case; this page covers hosts and systemd services.
What breaks at peak
Peak traffic grows every process at once. Workers open more connections, caches fill and request buffers stack up. The page cache shrinks first, then the kernel runs out of memory it can reclaim. It then scores each process by its memory use plus its oom_score_adj, and kills the highest.
The kernel logs the kill in two parts. The first line names the process that asked for memory, which is often not the one that dies:
php-fpm invoked oom-killer: gfp_mask=0x140cca(GFP_HIGHUSER_MOVABLE|__GFP_COMP), order=0, oom_score_adj=0
A table of every process and its RSS follows, then the verdict. A typical pair looks like this:
oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/system.slice/app.service,task=java,pid=4182,uid=1001
Out of memory: Killed process 4182 (java) total-vm:8123456kB, anon-rss:5234567kB, file-rss:1234kB, shmem-rss:0kB, UID:1001 pgtables:11234kB oom_score_adj:0
Kernels from 5.4 print that format. Kernel 4.19 printed Out of memory: Kill process 4182 (java) score 905 or sacrifice child, then a separate Killed process line. A kill inside a cgroup limit starts with Memory cgroup out of memory instead, and the constraint reads CONSTRAINT_MEMCG. The constraint field tells you which of the two problems you have: the whole host ran dry, or one service hit its own cap.
systemd adds its own lines for the unit that lost a process:
app.service: A process of this unit has been killed by the OOM killer.
app.service: Failed with result 'oom-kill'.
systemd 258 and older print the first line as shown. Newer releases word it The kernel OOM killer killed some processes in this unit.
Your users notice before you read any log. Requests in flight on the killed worker fail at once. If the victim was the database, every request fails until it restarts and recovers. A box with no swap often stalls for seconds before the kill, because the kernel throws away page cache and rereads your binaries from disk. That stall looks like a latency spike across the whole host.
The fix for the Linux OOM killer
Four changes, in the order I make them:
- Cap each service with
MemoryHighandMemoryMax. - Rank the processes with
OOMScoreAdjust. - Add a small swap file and lower swappiness.
- Leave overcommit at its default unless a component asks for a change.
Cap each service. On cgroup v2, systemd gives each service its own memory controller. MemoryHigh is a soft cap. Above it the kernel slows the service and reclaims its memory hard, but doesn’t kill it. MemoryMax is the hard cap. Past it the kernel runs the OOM killer inside that service only, and the rest of the host keeps running. Put both in a drop-in with systemctl edit app:
# /etc/systemd/system/app.service.d/memory.conf
[Service]
MemoryHigh=3G
MemoryMax=3584M
MemorySwapMax=512M
OOMPolicy=stop
OOMScoreAdjust=-500
Restart=on-failure
- Size
MemoryMaxfrom the service’s own peak. Measure its RSS at your expected peak and add 20 to 30 percent. Check that all your caps together leave room for the kernel and page cache. - Set
MemoryHigh10 to 15 percent belowMemoryMax. The gap gives the kernel room to reclaim before it kills. A service that sits atMemoryHighfor long periods runs slow, so the alert below watches pressure as well as kills. MemorySwapMaxcaps how much of the service may go to swap. Without it, one leaking service can fill the swap file and push the rest of the host into a stall.OOMPolicy=stoptells systemd to shut down the rest of the unit after a kill, andRestart=on-failurebrings it back. The default comes fromDefaultOOMPolicy, which isstopunless you changed it.killsetsmemory.oom.group, so the kernel ends every process in the unit together. I usekillfor apps with a master and workers, so a half-dead process tree doesn’t keep serving errors.
Restart the service after the edit. Check the caps took with systemctl show app -p MemoryHigh -p MemoryMax -p OOMPolicy, and read the live usage with cat /sys/fs/cgroup/system.slice/app.service/memory.current.
Rank the processes. oom_score_adj runs from -1000 to 1000 and shifts each process’s score. At -1000 the kernel never picks the process. Give your database and the app’s master process a negative value, and give batch jobs and report workers a positive one. Set it in the unit with OOMScoreAdjust= as above, or for a one-off command with choom -n 500 -- ./nightly-report. Leave -1000 for sshd and the few processes you need to log in and recover the box. If you protect too many processes, the kernel has nothing left to kill and the host locks up.
Add a little swap. A small swap file gives the kernel a place for memory your processes haven’t touched in hours, such as idle worker heaps. Without it, the kernel can only drop page cache, and it drops the pages your app reads next. On app hosts I add swap worth a few percent of RAM and set swappiness to 10. That keeps hot memory in RAM and turns a hard kill into a slow, visible decline you can alert on.
fallocate -l 2G /swapfile # use dd on file systems without fallocate support
chmod 600 /swapfile
mkswap /swapfile
swapon /swapfile
echo '/swapfile none swap defaults 0 0' >> /etc/fstab
# /etc/sysctl.d/90-memory.conf
vm.swappiness = 10
vm.swappiness sets how willing the kernel is to swap anonymous memory instead of dropping page cache. The default is 60. Skip swap on Kubernetes nodes unless you’ve configured the kubelet’s swap support, because the kubelet refuses to start with swap on by default.
Leave overcommit alone. vm.overcommit_memory = 0, the default, lets programs reserve more memory than exists and guesses when to refuse. Mode 2 refuses allocations past CommitLimit, which is swap plus RAM times vm.overcommit_ratio percent, 50 by default. It swaps the OOM killer for Cannot allocate memory errors inside your app, and most runtimes crash on those anyway. Redis wants mode 1 so a background save can fork. Redis 7.2 and later warn Memory overcommit must be enabled at startup if it isn’t, and older releases log WARNING overcommit_memory is set to 0. Set it on Redis hosts and keep maxmemory below the container limit, as the Redis memory recipe explains.
Distro differences. The kernel side matches on all of them. Logging and service management differ:
| Distro | Kernel log and OOM line | Memory caps | User-space OOM killer | Test tools |
|---|---|---|---|---|
| Ubuntu and Debian | journalctl -k. Ubuntu 24.04 ships systemd 255, Debian 12 ships 252 and Debian 13 ships 257, so all print the older systemd line. | systemd drop-ins in /etc/systemd/system/<unit>.service.d/ | systemd-oomd, a separate package. Run systemctl is-active systemd-oomd to see if it runs. | apt install stress-ng |
| RHEL, Rocky, AlmaLinux | journalctl -k. RHEL 9 ships systemd 252. | Same drop-ins. RHEL 9 boots cgroup v2 by default. | Check with systemctl is-active systemd-oomd. | stress-ng from AppStream or EPEL |
| Amazon Linux 2023 | journalctl -k. Ships systemd 252. | Same drop-ins | The repo carries systemd-oomd-defaults. Check whether systemd-oomd runs before you rely on the kernel alone. | dnf install stress-ng |
| Alpine | dmesg. OpenRC prints no unit line. | rc_cgroup_settings in /etc/conf.d/<service>, with lines such as memory.high and memory.max in bytes, on rc_cgroup_mode="unified" | earlyoom from the community repo | apk add stress-ng from community |
systemd-oomd acts on memory pressure and swap use, and kills a whole cgroup before the kernel runs out. If it runs on your host, its kills show in journalctl -u systemd-oomd, not in the kernel log. Look there first when a service dies with no Killed process line.
Test it on your system
Use a staging host with the same RAM, swap and service caps as production. Drive your app at its expected peak with a fixed arrival rate, then add memory pressure on the side until the host runs short. The k6 breakpoint recipe has the load script.
Start the pressure process with a high OOM score, so a correct setup kills it first:
choom -n 1000 -- stress-ng --vm 1 --vm-bytes 90% --vm-keep --timeout 10m
--vm-bytes 90% asks for 90 percent of available memory. Raise it in steps if the host holds up. While it runs, watch the kernel log, pressure and each service’s cgroup:
journalctl -k -f | grep -E 'invoked oom-killer|Killed process|Memory cgroup out of memory'
watch -n 2 'cat /proc/pressure/memory; free -m; \
cat /sys/fs/cgroup/system.slice/app.service/memory.current \
/sys/fs/cgroup/system.slice/app.service/memory.events'
/proc/pressure/memory shows the share of time tasks waited for memory. memory.events counts high, max and oom_kill events for the service. A rising high count means the soft cap is throttling it.
Write your pass criteria down before the run. A good set:
- The OOM killer picks
stress-ng, and your app and database keep their PIDs. - Your app’s error rate and p99 stay inside the budget you set, or degrade in a way you accept.
- Run a second test that pushes the app itself past
MemoryMax, for example with a request that loads a large export. The kill showsCONSTRAINT_MEMCG, systemd restarts the unit and the rest of the host carries on. - With swap on,
free -mshows swap in use but the host keeps answering, with no long stalls in the load test’s latency.
Run it once on your current settings first. That tells you which process the kernel picks today.
Keep it healthy
node_exporter covers all of this with collectors that are on by default. node_vmstat_oom_kill comes from the vmstat collector and counts kills since boot, including kills inside cgroups. The pressure collector needs kernel 4.20 or later with PSI enabled.
groups:
- name: linux-memory
rules:
- alert: HostOOMKill
expr: increase(node_vmstat_oom_kill[5m]) > 0
for: 0m
labels:
severity: critical
annotations:
summary: "OOM killer ended a process on {{ $labels.instance }}"
description: "Run journalctl -k | grep -E 'Killed process|constraint=' to see the victim and whether a cgroup cap or the whole host ran out. Raise that service's MemoryMax or find the process that grew."
- alert: HostMemoryAvailableLow
expr: node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes < 0.10
for: 10m
labels:
severity: warning
annotations:
summary: "{{ $labels.instance }} has under 10% of memory available"
description: "The OOM killer runs next if this keeps falling. Check per-service use with systemd-cgtop -m and compare it with each unit's MemoryMax."
- alert: HostMemoryPressure
expr: rate(node_pressure_memory_waiting_seconds_total[5m]) > 0.20
for: 10m
labels:
severity: warning
annotations:
summary: "Tasks on {{ $labels.instance }} wait on memory over 20% of the time"
description: "Services are stalling in reclaim. Look for units at MemoryHigh in memory.events, then raise the cap, add RAM or move a workload off the host."
- alert: HostSwapNearlyFull
expr: (1 - node_memory_SwapFree_bytes / node_memory_SwapTotal_bytes) > 0.80 and node_memory_SwapTotal_bytes > 0
for: 15m
labels:
severity: warning
annotations:
summary: "Swap on {{ $labels.instance }} is over 80% used"
description: "Swap is meant for idle pages, so a full swap file means something is leaking. Find it with smem or systemd-cgtop, and set MemorySwapMax on that unit."
The 10 percent and 20 percent thresholds are starting points. Watch a normal peak first, then set each one a little past what a healthy day reaches. If you run vm.overcommit_memory = 2, add a rule on node_memory_Committed_AS_bytes / node_memory_CommitLimit_bytes above 0.9, since allocations start failing at 1.0.
Roll back
Save the current values first:
systemctl show app -p MemoryHigh -p MemoryMax -p MemorySwapMax -p OOMPolicy -p OOMScoreAdjust > /root/app-memory-before.txt
sysctl vm.swappiness vm.overcommit_memory > /root/sysctl-memory-before.txt
swapon --show >> /root/sysctl-memory-before.txt
To undo the service caps, remove the drop-in and restart:
rm /etc/systemd/system/app.service.d/memory.conf
systemctl daemon-reload
systemctl restart app
To undo swap, run swapoff /swapfile, delete its line from /etc/fstab and remove the file. swapoff has to pull every swapped page back into RAM, so run it while the host has free memory. Remove /etc/sysctl.d/90-memory.conf and load the saved values with sysctl -p /root/sysctl-memory-before.txt.
If a cap is killing a healthy service under normal load, raise MemoryMax now with systemctl set-property app MemoryMax=4G, which applies without a restart. Then fix the drop-in. Other reverts can wait for a quiet hour, since a restart drops in-flight requests.
More recipes
- Alpine DNS Issues, musl Timeouts and Kubernetes ndots
Lower Kubernetes ndots, cache DNS on each node and use full names, so the musl resolver in Alpine images stops failing lookups at peak.
- Tuned Profiles on RHEL, and When They Override Sysctls
Check which tuned profile runs, keep sysctls in /etc/sysctl.d and carry the rest in a custom profile, so a reboot or tuned restart can't undo your tuning.
- Possible SYN Flooding on Port, Monitor the SYN Backlog
Alert on SYN cookie and listen queue counters and size the listen backlog, not tcp_max_syn_backlog, so peak traffic stops tripping SYN flood warnings.
- Systemd LimitNOFILE, Fix Open File Limits for Services
limits.conf isn't applied to a systemd service, so set LimitNOFILE in a drop-in, restart the unit and read the new limit back from /proc.