tuned profiles on RHEL decide a host’s swappiness, dirty-page limits, CPU governor and disk readahead before your app starts. RHEL, Rocky Linux and AlmaLinux run the tuned service on standard installs, and it applies a profile at every boot. If you tuned the box by hand, find out which of your values tuned keeps and which it replaces.
Run this first:
tuned-adm active
tuned-adm recommend
tuned-adm active prints Current active profile: virtual-guest or similar. tuned-adm recommend prints the profile tuned would pick for this hardware. On a cloud VM, both tend to say virtual-guest.
What breaks at peak
The stock server profiles change real limits. These are from TuneD’s own profile files:
| Profile | Picked by default for | What it sets |
|---|---|---|
throughput-performance | Bare-metal RHEL server installs | CPU governor performance, vm.swappiness=10, dirty limits at 40% and 10% of memory, disk readahead of at least 4096 KiB, net.core.somaxconn raised to 2048 if lower |
virtual-guest | Any VM, cloud instances included | Everything in throughput-performance, with the dirty limit at 30% and vm.swappiness=30 |
latency-performance | Nothing; you pick it | CPU governor performance, C-state limits, vm.swappiness=10, dirty limits at 10% and 3% |
network-latency | Nothing; you pick it | latency-performance plus transparent huge pages never, busy polling and TCP Fast Open |
balanced | Hosts that match no other rule | Power-saving CPU governors |
Problems show up in three ways:
- A value you set by hand disappears. You ran
sysctl -w net.core.somaxconn=8192during an incident and never wrote it to a file. tuned reapplies its profile at boot, onsystemctl restart tunedand on everytuned-adm profilecall. Your runtime value is gone and the listen queue overflows again at the next peak. - A non-sysctl setting changes under you. tuned sets transparent huge pages, the CPU governor and readahead through its own plugins. A systemd unit or script that sets those runs in a race with tuned. Whoever writes last wins, and that can change from one boot to the next.
- A package’s sysctl loses. Files in
/usr/lib/sysctl.d/belong to packages. tuned’s reapply step skips that directory, so if a profile sets the same key, the profile wins.
tuned-adm verify makes it worse when someone follows its hint. It prints this when your sysctls differ from the profile:
Verification failed, current system settings differ from the preset profile.
You can mostly fix this by restarting the TuneD daemon, e.g.:
systemctl restart tuned
Restarting tuned wipes every runtime-only value on the box. It does that on a production host at the worst moment.
When tuned overrides sysctl, and the fix
tuned’s sysctl plugin has a safety net. /etc/tuned/tuned-main.conf ships with reapply_sysctl = 1. After tuned applies its own sysctls, it reloads the files in /run/sysctl.d/, /etc/sysctl.d/ and /etc/sysctl.conf on top. So a key in /etc/sysctl.d/ beats the profile. tuned overrides a sysctl only in these cases:
- You set it at runtime and never wrote it to a file.
- It lives in
/usr/lib/sysctl.d/or another directory outside those three. - Someone set
reapply_sysctl = 0. - It isn’t a sysctl at all, such as transparent huge pages or the CPU governor.
The fix follows from that list.
Keep your sysctls in /etc/sysctl.d. Write every kernel setting you depend on to a file there and load it with sysctl --system. The network tuning recipe uses /etc/sysctl.d/90-web-server.conf for this reason. List the order systemd loads the files in with:
systemd-analyze cat-config sysctl.d | grep -v '^#' | grep -v '^$'
grep -rn 'somaxconn\|swappiness' /usr/lib/tuned/ /etc/tuned/
The second line finds which tuned profiles set a key you care about.
Put the rest in a custom profile. A custom profile includes a stock one and changes only what you list. tuned applies it at boot along with everything else, so no race remains:
# /etc/tuned/profiles/web-server/tuned.conf (TuneD 2.23 and later)
# /etc/tuned/web-server/tuned.conf (TuneD 2.22 and earlier)
[main]
summary=virtual-guest with web server settings
include=virtual-guest
[vm]
transparent_hugepages=madvise
[sysctl]
vm.swappiness=10
net.core.somaxconn=4096
tuned-adm profile web-server
tuned-adm active
include=virtual-guestkeeps the stock profile’s settings. Pick the profile thattuned-adm recommendnames for the host.[vm] transparent_hugepages=madvisesets THP. tuned ignores it whentransparent_hugepage=is on the kernel command line, so use one method or the other. The transparent huge pages recipe explains which value to pick.[sysctl]sets kernel keys. A value starting with>raises a key only if the current value is lower, as the stock profiles do withsomaxconn=>2048.
Check where your TuneD version keeps profiles with rpm -q tuned and ls /usr/lib/tuned/. TuneD 2.23 moved stock profiles to /usr/lib/tuned/profiles/ and custom ones to /etc/tuned/profiles/.
Protect runtime sysctls on Kubernetes nodes. CNI plugins and kube-proxy set sysctls such as net.ipv4.ip_forward at runtime. With reapply_sysctl = 1, a tuned restart reloads /etc/sysctl.d/ over them. TuneD 2.27 added reapply_sysctl_exclude to tuned-main.conf for this:
# /etc/tuned/tuned-main.conf (changed lines only)
reapply_sysctl_exclude = net.ipv4.ip_forward, net.ipv4.conf.*.forwarding, net.netfilter.nf_conntrack_max
On older TuneD, keep /etc/sysctl.d/ free of keys your CNI or kube-proxy manage.
Keep tuned running. You can stop it with tuned-adm off and systemctl disable --now tuned. I keep it and use a custom profile instead. Turning it off also drops the CPU governor and readahead it set, and someone chose those for a reason. A custom profile also leaves a file that the next engineer can read.
Distro differences. TuneD exists outside the Red Hat family, but seldom runs there by default:
| Distro | Package | State on a standard install | Profile directory |
|---|---|---|---|
| Ubuntu and Debian | tuned; 2.21 on Ubuntu 24.04, 2.25.1 on Debian 13 | Not installed. Sysctls come from /etc/sysctl.d/ alone. | Depends on the version, see above |
| RHEL, Rocky, AlmaLinux | tuned | Installed and running. Check with systemctl is-active tuned. | Depends on the version, see above |
| Amazon Linux 2023 | tuned 2.25.1 in the AL2023 repository | Check with systemctl is-active tuned on your AMI | /usr/lib/tuned/profiles/ and /etc/tuned/profiles/ |
| Alpine | tuned and tuned-openrc in the edge community repository only | Not installed | Depends on the version, see above |
Test it on your system
The test proves your settings survive every event that reapplies a profile. Run it on a staging host built from the same image as production.
Record the values you care about, then trigger each event and compare:
KEYS="vm.swappiness vm.dirty_ratio vm.dirty_bytes net.core.somaxconn net.ipv4.tcp_max_syn_backlog"
snap() { sysctl $KEYS; cat /sys/kernel/mm/transparent_hugepage/enabled; tuned-adm active; }
snap > before.txt
systemctl restart tuned && snap > after-restart.txt
tuned-adm profile "$(tuned-adm active | awk -F': ' '{print $2}')" && snap > after-reselect.txt
diff before.txt after-restart.txt; diff before.txt after-reselect.txt
Then reboot and run snap once more. A pass is no diff in any of the three. After that, run your normal load test at your expected peak and watch the counters your tuning targets. For somaxconn, that means nstat -az TcpExtListenOverflows stays flat.
Check the tuned log for the settings it applied and any it skipped:
grep -E 'reapplying system sysctl|ignoring value from profile' /var/log/tuned/tuned.log
Keep it healthy
node_exporter’s sysctl collector reads any key you name. It ships in node_exporter 1.4 and later and is off by default:
node_exporter --collector.sysctl \
--collector.sysctl.include=vm.swappiness \
--collector.sysctl.include=net.core.somaxconn \
--collector.textfile.directory=/var/lib/node_exporter/textfile
That exports node_sysctl_vm_swappiness and node_sysctl_net_core_somaxconn. For the profile name, write a textfile metric from cron or a systemd timer:
#!/bin/sh
p=$(tuned-adm active 2>/dev/null | sed -n 's/^Current active profile: //p')
echo "tuned_active_profile{profile=\"${p:-none}\"} 1" > /var/lib/node_exporter/textfile/tuned.prom.$$
mv /var/lib/node_exporter/textfile/tuned.prom.$$ /var/lib/node_exporter/textfile/tuned.prom
The rules below assume your custom profile is web-server and your target somaxconn is 4096. Change both to yours:
groups:
- name: linux-tuned
rules:
- alert: TunedProfileDrift
expr: tuned_active_profile{profile!="web-server", role="web"} == 1
for: 15m
labels:
severity: warning
annotations:
summary: "{{ $labels.instance }} runs tuned profile {{ $labels.profile }}"
description: "Someone switched profiles or a new image shipped without yours. Run tuned-adm profile web-server, then check /var/log/tuned/tuned.log."
- alert: SomaxconnBelowTarget
expr: node_sysctl_net_core_somaxconn{role="web"} < 4096
for: 15m
labels:
severity: warning
annotations:
summary: "net.core.somaxconn on {{ $labels.instance }} is {{ $value }}"
description: "A tuned restart or a runtime-only change reset it. Put the value in /etc/sysctl.d/ or your tuned profile, then run sysctl --system."
Add the role label to your node targets in the scrape config. Add one rule per key you can’t afford to lose, and skip the rest. The alert carries no peak-time noise, since these values only change when someone or something writes them.
Roll back
Save the starting state first:
tuned-adm active > /root/tuned-before.txt
cp /etc/tuned/tuned-main.conf /root/tuned-main.conf.before
sysctl -a 2>/dev/null > /root/sysctl-all-before.txt
To roll back, switch to the old profile with tuned-adm profile virtual-guest, or whichever name you saved. tuned restores the values the custom profile replaced when it unloads it. Restore tuned-main.conf from the copy and restart tuned. Delete the custom profile directory once nothing uses it.
If the change broke something on a live host, such as networking after a sysctl exclusion went wrong, switch the profile back now. A profile switch restarts no services, so it carries little risk at peak. Restarting tuned reloads every sysctl file over any runtime values, so leave that for a quiet window.
More recipes
- Alpine DNS Issues, musl Timeouts and Kubernetes ndots
Lower Kubernetes ndots, cache DNS on each node and use full names, so the musl resolver in Alpine images stops failing lookups at peak.
- Out of Memory Killed Process on Linux, Fix the OOM Killer
Cap each service's memory, add a little swap and set OOM scores, so the Linux OOM killer stops the process you can lose instead of your app.
- Possible SYN Flooding on Port, Monitor the SYN Backlog
Alert on SYN cookie and listen queue counters and size the listen backlog, not tcp_max_syn_backlog, so peak traffic stops tripping SYN flood warnings.
- Systemd LimitNOFILE, Fix Open File Limits for Services
limits.conf isn't applied to a systemd service, so set LimitNOFILE in a drop-in, restart the unit and read the new limit back from /proc.