Tuned Profiles on RHEL, and When They Override Sysctls

Check which tuned profile runs, keep sysctls in /etc/sysctl.d and carry the rest in a custom profile, so a reboot or tuned restart can't undo your tuning.

Published

Applies to TuneD 2.10 and later on RHEL 8, 9 and 10, Rocky Linux and AlmaLinux; reapply_sysctl_exclude needs TuneD 2.27

tuned profiles on RHEL decide a host’s swappiness, dirty-page limits, CPU governor and disk readahead before your app starts. RHEL, Rocky Linux and AlmaLinux run the tuned service on standard installs, and it applies a profile at every boot. If you tuned the box by hand, find out which of your values tuned keeps and which it replaces.

Run this first:

tuned-adm active
tuned-adm recommend

tuned-adm active prints Current active profile: virtual-guest or similar. tuned-adm recommend prints the profile tuned would pick for this hardware. On a cloud VM, both tend to say virtual-guest.

What breaks at peak

The stock server profiles change real limits. These are from TuneD’s own profile files:

ProfilePicked by default forWhat it sets
throughput-performanceBare-metal RHEL server installsCPU governor performance, vm.swappiness=10, dirty limits at 40% and 10% of memory, disk readahead of at least 4096 KiB, net.core.somaxconn raised to 2048 if lower
virtual-guestAny VM, cloud instances includedEverything in throughput-performance, with the dirty limit at 30% and vm.swappiness=30
latency-performanceNothing; you pick itCPU governor performance, C-state limits, vm.swappiness=10, dirty limits at 10% and 3%
network-latencyNothing; you pick itlatency-performance plus transparent huge pages never, busy polling and TCP Fast Open
balancedHosts that match no other rulePower-saving CPU governors

Problems show up in three ways:

  • A value you set by hand disappears. You ran sysctl -w net.core.somaxconn=8192 during an incident and never wrote it to a file. tuned reapplies its profile at boot, on systemctl restart tuned and on every tuned-adm profile call. Your runtime value is gone and the listen queue overflows again at the next peak.
  • A non-sysctl setting changes under you. tuned sets transparent huge pages, the CPU governor and readahead through its own plugins. A systemd unit or script that sets those runs in a race with tuned. Whoever writes last wins, and that can change from one boot to the next.
  • A package’s sysctl loses. Files in /usr/lib/sysctl.d/ belong to packages. tuned’s reapply step skips that directory, so if a profile sets the same key, the profile wins.

tuned-adm verify makes it worse when someone follows its hint. It prints this when your sysctls differ from the profile:

Verification failed, current system settings differ from the preset profile.
You can mostly fix this by restarting the TuneD daemon, e.g.:
  systemctl restart tuned

Restarting tuned wipes every runtime-only value on the box. It does that on a production host at the worst moment.

When tuned overrides sysctl, and the fix

tuned’s sysctl plugin has a safety net. /etc/tuned/tuned-main.conf ships with reapply_sysctl = 1. After tuned applies its own sysctls, it reloads the files in /run/sysctl.d/, /etc/sysctl.d/ and /etc/sysctl.conf on top. So a key in /etc/sysctl.d/ beats the profile. tuned overrides a sysctl only in these cases:

  • You set it at runtime and never wrote it to a file.
  • It lives in /usr/lib/sysctl.d/ or another directory outside those three.
  • Someone set reapply_sysctl = 0.
  • It isn’t a sysctl at all, such as transparent huge pages or the CPU governor.

The fix follows from that list.

Keep your sysctls in /etc/sysctl.d. Write every kernel setting you depend on to a file there and load it with sysctl --system. The network tuning recipe uses /etc/sysctl.d/90-web-server.conf for this reason. List the order systemd loads the files in with:

systemd-analyze cat-config sysctl.d | grep -v '^#' | grep -v '^$'
grep -rn 'somaxconn\|swappiness' /usr/lib/tuned/ /etc/tuned/

The second line finds which tuned profiles set a key you care about.

Put the rest in a custom profile. A custom profile includes a stock one and changes only what you list. tuned applies it at boot along with everything else, so no race remains:

# /etc/tuned/profiles/web-server/tuned.conf   (TuneD 2.23 and later)
# /etc/tuned/web-server/tuned.conf            (TuneD 2.22 and earlier)
[main]
summary=virtual-guest with web server settings
include=virtual-guest

[vm]
transparent_hugepages=madvise

[sysctl]
vm.swappiness=10
net.core.somaxconn=4096
tuned-adm profile web-server
tuned-adm active
  • include=virtual-guest keeps the stock profile’s settings. Pick the profile that tuned-adm recommend names for the host.
  • [vm] transparent_hugepages=madvise sets THP. tuned ignores it when transparent_hugepage= is on the kernel command line, so use one method or the other. The transparent huge pages recipe explains which value to pick.
  • [sysctl] sets kernel keys. A value starting with > raises a key only if the current value is lower, as the stock profiles do with somaxconn=>2048.

Check where your TuneD version keeps profiles with rpm -q tuned and ls /usr/lib/tuned/. TuneD 2.23 moved stock profiles to /usr/lib/tuned/profiles/ and custom ones to /etc/tuned/profiles/.

Protect runtime sysctls on Kubernetes nodes. CNI plugins and kube-proxy set sysctls such as net.ipv4.ip_forward at runtime. With reapply_sysctl = 1, a tuned restart reloads /etc/sysctl.d/ over them. TuneD 2.27 added reapply_sysctl_exclude to tuned-main.conf for this:

# /etc/tuned/tuned-main.conf (changed lines only)
reapply_sysctl_exclude = net.ipv4.ip_forward, net.ipv4.conf.*.forwarding, net.netfilter.nf_conntrack_max

On older TuneD, keep /etc/sysctl.d/ free of keys your CNI or kube-proxy manage.

Keep tuned running. You can stop it with tuned-adm off and systemctl disable --now tuned. I keep it and use a custom profile instead. Turning it off also drops the CPU governor and readahead it set, and someone chose those for a reason. A custom profile also leaves a file that the next engineer can read.

Distro differences. TuneD exists outside the Red Hat family, but seldom runs there by default:

DistroPackageState on a standard installProfile directory
Ubuntu and Debiantuned; 2.21 on Ubuntu 24.04, 2.25.1 on Debian 13Not installed. Sysctls come from /etc/sysctl.d/ alone.Depends on the version, see above
RHEL, Rocky, AlmaLinuxtunedInstalled and running. Check with systemctl is-active tuned.Depends on the version, see above
Amazon Linux 2023tuned 2.25.1 in the AL2023 repositoryCheck with systemctl is-active tuned on your AMI/usr/lib/tuned/profiles/ and /etc/tuned/profiles/
Alpinetuned and tuned-openrc in the edge community repository onlyNot installedDepends on the version, see above

Test it on your system

The test proves your settings survive every event that reapplies a profile. Run it on a staging host built from the same image as production.

Record the values you care about, then trigger each event and compare:

KEYS="vm.swappiness vm.dirty_ratio vm.dirty_bytes net.core.somaxconn net.ipv4.tcp_max_syn_backlog"
snap() { sysctl $KEYS; cat /sys/kernel/mm/transparent_hugepage/enabled; tuned-adm active; }
snap > before.txt
systemctl restart tuned && snap > after-restart.txt
tuned-adm profile "$(tuned-adm active | awk -F': ' '{print $2}')" && snap > after-reselect.txt
diff before.txt after-restart.txt; diff before.txt after-reselect.txt

Then reboot and run snap once more. A pass is no diff in any of the three. After that, run your normal load test at your expected peak and watch the counters your tuning targets. For somaxconn, that means nstat -az TcpExtListenOverflows stays flat.

Check the tuned log for the settings it applied and any it skipped:

grep -E 'reapplying system sysctl|ignoring value from profile' /var/log/tuned/tuned.log

Keep it healthy

node_exporter’s sysctl collector reads any key you name. It ships in node_exporter 1.4 and later and is off by default:

node_exporter --collector.sysctl \
  --collector.sysctl.include=vm.swappiness \
  --collector.sysctl.include=net.core.somaxconn \
  --collector.textfile.directory=/var/lib/node_exporter/textfile

That exports node_sysctl_vm_swappiness and node_sysctl_net_core_somaxconn. For the profile name, write a textfile metric from cron or a systemd timer:

#!/bin/sh
p=$(tuned-adm active 2>/dev/null | sed -n 's/^Current active profile: //p')
echo "tuned_active_profile{profile=\"${p:-none}\"} 1" > /var/lib/node_exporter/textfile/tuned.prom.$$
mv /var/lib/node_exporter/textfile/tuned.prom.$$ /var/lib/node_exporter/textfile/tuned.prom

The rules below assume your custom profile is web-server and your target somaxconn is 4096. Change both to yours:

groups:
  - name: linux-tuned
    rules:
      - alert: TunedProfileDrift
        expr: tuned_active_profile{profile!="web-server", role="web"} == 1
        for: 15m
        labels:
          severity: warning
        annotations:
          summary: "{{ $labels.instance }} runs tuned profile {{ $labels.profile }}"
          description: "Someone switched profiles or a new image shipped without yours. Run tuned-adm profile web-server, then check /var/log/tuned/tuned.log."
      - alert: SomaxconnBelowTarget
        expr: node_sysctl_net_core_somaxconn{role="web"} < 4096
        for: 15m
        labels:
          severity: warning
        annotations:
          summary: "net.core.somaxconn on {{ $labels.instance }} is {{ $value }}"
          description: "A tuned restart or a runtime-only change reset it. Put the value in /etc/sysctl.d/ or your tuned profile, then run sysctl --system."

Add the role label to your node targets in the scrape config. Add one rule per key you can’t afford to lose, and skip the rest. The alert carries no peak-time noise, since these values only change when someone or something writes them.

Roll back

Save the starting state first:

tuned-adm active > /root/tuned-before.txt
cp /etc/tuned/tuned-main.conf /root/tuned-main.conf.before
sysctl -a 2>/dev/null > /root/sysctl-all-before.txt

To roll back, switch to the old profile with tuned-adm profile virtual-guest, or whichever name you saved. tuned restores the values the custom profile replaced when it unloads it. Restore tuned-main.conf from the copy and restart tuned. Delete the custom profile directory once nothing uses it.

If the change broke something on a live host, such as networking after a sysctl exclusion went wrong, switch the profile back now. A profile switch restarts no services, so it carries little risk at peak. Restarting tuned reloads every sysctl file over any runtime values, so leave that for a quiet window.

More recipes

Further reading