To disable transparent huge pages, set /sys/kernel/mm/transparent_hugepage/enabled to madvise or never, and make that stick across reboots. With THP on always, the kernel backs your processes’ memory with 2 MiB pages. Redis and several databases pay for that in latency and memory. Some distros ship always and some ship madvise, so check yours before you change anything:
cat /sys/kernel/mm/transparent_hugepage/enabled
cat /sys/kernel/mm/transparent_hugepage/defrag
The value in brackets is the active one. [always] madvise never means THP is on for every process.
What breaks at peak
A huge page covers 2 MiB where a normal page covers 4 KiB. That saves TLB lookups for programs that walk large, dense memory. A cache or database that touches small, scattered keys gets the costs instead.
- Fork and copy-on-write. Redis forks for every RDB snapshot, AOF rewrite and full replica sync. After the fork, each write to a shared page makes the kernel copy that page. With THP, one small write copies 2 MiB instead of 4 KiB. A busy instance during
BGSAVEuses far more memory than its dataset, and each copy adds latency to the write that caused it. - Page faults that stall. The first touch of a 2 MiB region has to find and zero a whole huge page. If memory is fragmented and
defragallows it, the process stalls while the kernel compacts memory to build one. Those stalls land on whichever request touched new memory, so they show up as p99 spikes with idle CPU. - Memory bloat. A huge page counts as fully used even when the process wrote one byte of it. Allocators that free memory in small pieces can’t hand partial huge pages back, so RSS grows and stays high.
Redis checks the setting at startup. Current releases log this line when THP is always:
WARNING You have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo madvise > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled (set to 'madvise' or 'never').
Older releases, 7.0.0 among them, begin with WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. in lowercase, and the rest matches. Since Redis 6.2, disable-thp yes is the default, and Redis turns THP off for its own process with prctl(PR_SET_THP_DISABLE). It logs the warning only when that call fails. So on a current Redis, a missing warning doesn’t prove the host is safe for the other processes on it.
In production it shows up as slow requests during snapshots and replica syncs. It also shows up as a cache OOM-killed during a save it survived the week before.
The fix, disable transparent huge pages with madvise
I set enabled to madvise rather than never. madvise gives huge pages only to memory regions whose program asks with madvise(MADV_HUGEPAGE). Redis and most databases don’t ask, so they get normal pages. A JVM started with -XX:+UseTransparentHugePages asks, and keeps its huge pages. never turns THP off for all processes, including programs that benefit.
Leave defrag at madvise, the kernel’s default. Then only regions that asked for huge pages will stall to compact memory.
Apply it now. The change takes effect at once for new allocations. Pages that are already huge stay huge until the process frees them, so restart the processes you care about afterwards:
echo madvise > /sys/kernel/mm/transparent_hugepage/enabled
echo madvise > /sys/kernel/mm/transparent_hugepage/defrag
systemctl restart redis-server
Make it stick with the kernel command line. A boot parameter sets the value before any process starts:
transparent_hugepage=madvise
On RHEL and Amazon Linux 2023, add it with grubby --update-kernel=ALL --args="transparent_hugepage=madvise". On Ubuntu and Debian, append it to GRUB_CMDLINE_LINUX_DEFAULT in /etc/default/grub and run update-grub. Reboot, then check /proc/cmdline and the sysfs file.
Or use a systemd unit. If you can’t reboot soon, or you don’t control the bootloader, a oneshot unit sets it early in boot:
# /etc/systemd/system/thp-madvise.service
[Unit]
Description=Set transparent huge pages to madvise
DefaultDependencies=no
After=sysinit.target local-fs.target
Before=basic.target redis-server.service
[Service]
Type=oneshot
ExecStart=/bin/sh -c 'echo madvise > /sys/kernel/mm/transparent_hugepage/enabled; echo madvise > /sys/kernel/mm/transparent_hugepage/defrag'
[Install]
WantedBy=basic.target
Enable it with systemctl daemon-reload && systemctl enable --now thp-madvise. Replace redis-server.service with your database’s unit name. I use the command line on hosts I build and the unit on hosts I inherit.
Watch for tuned. On RHEL and its rebuilds, tuned can set THP from the [vm] section of its active profile. That can undo a unit like the one above. tuned skips that setting when transparent_hugepage= is already on the kernel command line. That makes the boot parameter the safer choice there. The tuned profiles recipe shows how to carry the setting in a custom profile instead.
Keep Redis on its own setting too. Leave disable-thp yes in redis.conf. It protects Redis on a host where someone sets THP back to always. Check the process itself:
grep THP_enabled /proc/$(pidof redis-server)/status # kernel 5.0 and later; 0 means off
grep AnonHugePages /proc/$(pidof redis-server)/smaps_rollup
MongoDB 8.0 wants the opposite. MongoDB 8.0 moved to a new TCMalloc and now asks for THP on: enabled set to always, defrag set to defer+madvise and khugepaged/max_ptes_none set to 0. MongoDB 7.0 and older still tell you to disable it. THP is a host setting, so don’t run MongoDB 8.0 on the same host as Redis. Check your database vendor’s current docs before you pick a value, since they disagree. For MongoDB’s other memory setting in containers, read the WiredTiger cache and maxPoolSize recipe.
Kubernetes nodes. Containers share the node’s kernel, so a pod can’t change THP for itself. Set it in the node image or user data, and give Redis and MongoDB 8.0 their own node pools if both run in one cluster.
Distro differences. Defaults and tools differ:
| Distro | Default enabled | Boot parameter goes in | Notes |
|---|---|---|---|
| Ubuntu | madvise on Ubuntu kernels since 2017 | GRUB_CMDLINE_LINUX_DEFAULT in /etc/default/grub, then update-grub | Already safe for Redis by default. Check after you install a non-Ubuntu kernel. |
| Debian | always on Debian 12 and 13 kernels | Same as Ubuntu | Needs the change |
| RHEL, Rocky, AlmaLinux | always | grubby --update-kernel=ALL --args=... | tuned can override sysfs writes. Use the boot parameter or a tuned profile. |
| Amazon Linux 2023 | AWS doesn’t document it. Read the sysfs file on your AMI. | grubby --update-kernel=ALL --args=... | Bake it into your AMI or launch template user data |
| Alpine | always on the linux-lts and linux-virt kernels, the upstream default | default_kernel_opts in /etc/update-extlinux.conf, then update-extlinux | In containers, the host’s setting applies |
Test it on your system
Use a staging host with production’s kernel, memory size and Redis or database version. Run the test twice, once with enabled on always and once on madvise, with a restart of the service between them. On Redis, set disable-thp no for the always run, or the process ignores the host setting.
Drive Redis with a fixed write-heavy load, and trigger a save in the middle:
memtier_benchmark -s redis-staging -p 6379 --protocol=redis \
--ratio=1:1 --data-size-range=100-2000 --key-pattern=R:R \
--key-maximum=5000000 --rate-limiting=500 \
--clients=20 --threads=4 --test-time=600 &
sleep 120 && redis-cli -h redis-staging BGSAVE
--rate-limiting caps requests per second per connection, so the load stays fixed while latency changes. For a database, use your usual load tool with a fixed arrival rate, such as k6’s constant-arrival-rate.
While it runs, watch fork time, memory and compaction:
watch -n 2 "redis-cli -h redis-staging INFO stats | grep latest_fork_usec; \
redis-cli -h redis-staging INFO memory | grep -E '^used_memory_rss_human'; \
grep -E '^(compact_stall|thp_fault_alloc|thp_collapse_alloc) ' /proc/vmstat; \
grep AnonHugePages /proc/meminfo"
latest_fork_usec is how long the last fork blocked Redis. compact_stall counts the times a process stopped to compact memory. thp_fault_alloc counts huge pages handed out on page faults.
Set your pass criteria before you start. On the madvise run, p99 during BGSAVE stays inside your latency budget. RSS stays below your container limit during the save, compact_stall stays flat, and AnonHugePages for Redis reads 0 kB.
Keep it healthy
node_exporter exports node_memory_AnonHugePages_bytes from its default meminfo collector. Its vmstat collector skips the THP and compaction counters by default, so extend the field list:
node_exporter --collector.vmstat.fields='^(oom_kill|pgpg|pswp|pg.*fault|compact_stall|thp_fault_alloc).*'
Those fields arrive as node_vmstat_compact_stall and node_vmstat_thp_fault_alloc. node_exporter can’t read /sys/kernel/mm, so add a small textfile collector script for the setting itself. Run it from cron or a systemd timer:
#!/bin/sh
# writes thp_enabled{mode="..."} 1 for node_exporter's textfile collector
mode=$(sed -n 's/.*\[\(.*\)\].*/\1/p' /sys/kernel/mm/transparent_hugepage/enabled)
printf 'thp_enabled{mode="%s"} 1\n' "$mode" > /var/lib/node_exporter/textfile/thp.prom.$$
mv /var/lib/node_exporter/textfile/thp.prom.$$ /var/lib/node_exporter/textfile/thp.prom
Point --collector.textfile.directory at /var/lib/node_exporter/textfile. Then alert on drift and on stalls:
groups:
- name: linux-thp
rules:
- alert: THPAlwaysOnDatabaseHost
expr: thp_enabled{mode="always", role=~"redis|database"} == 1
for: 15m
labels:
severity: warning
annotations:
summary: "Transparent huge pages are set to always on {{ $labels.instance }}"
description: "Check /proc/cmdline and tuned-adm active, set transparent_hugepage=madvise on the kernel command line and restart the database. Skip hosts that run MongoDB 8.0 or later."
- alert: MemoryCompactionStalls
expr: rate(node_vmstat_compact_stall[5m]) > 1
for: 10m
labels:
severity: warning
annotations:
summary: "Processes on {{ $labels.instance }} are stalling to compact memory"
description: "Direct compaction adds latency to page faults. Check that defrag is madvise, then look for processes with large AnonHugePages in /proc/<pid>/smaps_rollup."
The role label comes from your scrape config, so add it to Redis and database targets. Set the compaction threshold from a normal day’s rate. A few stalls an hour is common on a host that runs a JVM with huge pages. A steady rate during peak means some process is paying for it.
Roll back
Save the current values first:
cat /sys/kernel/mm/transparent_hugepage/enabled /sys/kernel/mm/transparent_hugepage/defrag > /root/thp-before.txt
cat /proc/cmdline >> /root/thp-before.txt
To undo the runtime change, write the old values back, for example echo always > /sys/kernel/mm/transparent_hugepage/enabled. To undo the boot parameter, run grubby --update-kernel=ALL --remove-args="transparent_hugepage" on RHEL and Amazon Linux, or remove it from /etc/default/grub and run update-grub. Disable and delete thp-madvise.service if you added it.
Roll back now if a workload that relies on huge pages slows down after the change. A JVM or a MongoDB 8.0 host you missed are the likely cases. Otherwise schedule it for a quiet hour, because the database restart that follows drops its connections.
More recipes
- Alpine DNS Issues, musl Timeouts and Kubernetes ndots
Lower Kubernetes ndots, cache DNS on each node and use full names, so the musl resolver in Alpine images stops failing lookups at peak.
- Out of Memory Killed Process on Linux, Fix the OOM Killer
Cap each service's memory, add a little swap and set OOM scores, so the Linux OOM killer stops the process you can lose instead of your app.
- Tuned Profiles on RHEL, and When They Override Sysctls
Check which tuned profile runs, keep sysctls in /etc/sysctl.d and carry the rest in a custom profile, so a reboot or tuned restart can't undo your tuning.
- Possible SYN Flooding on Port, Monitor the SYN Backlog
Alert on SYN cookie and listen queue counters and size the listen backlog, not tcp_max_syn_backlog, so peak traffic stops tripping SYN flood warnings.