Alpine DNS issues in Kubernetes come from two things meeting: the musl C library’s resolver and the cluster’s ndots:5 default. Each lookup for an outside name turns into several queries, and musl gives up on the whole lookup when any one of them fails. At low traffic you never notice. At peak, CoreDNS gets busy, one query times out, and your app logs getaddrinfo EAI_AGAIN.
The fix is fewer queries per lookup and a resolver path with fewer places to drop a packet. If the conntrack table is filling as well, read the conntrack section of Linux network tuning for web servers.
What breaks at peak
Kubernetes writes this into each pod’s /etc/resolv.conf, plus any search domains the node has:
nameserver 10.96.0.10
search shop.svc.cluster.local svc.cluster.local cluster.local
options ndots:5
ndots:5 means any name with fewer than five dots goes through the search list first. api.stripe.com has two dots. The resolver asks for api.stripe.com.shop.svc.cluster.local first, then the next two suffixes, then the name itself. musl sends the A and AAAA queries together, so one lookup costs up to eight queries. Three of every four come back NXDOMAIN.
musl’s resolver differs from glibc’s in three ways that matter here. I checked each against musl’s source:
- It stops at the first failure that isn’t NXDOMAIN. A timeout or
SERVFAILon any search suffix ends the lookup withEAI_AGAIN. glibc notes theSERVFAILand moves on to the next name. - It reads three options. musl honors
ndots,timeoutandattemptsand ignores the rest. glibc’ssingle-request-reopen,use-vcandrotatedo nothing on Alpine. - It had no DNS over TCP before musl 1.2.4. Alpine 3.18 shipped 1.2.4. On older images, an answer too big for UDP fails instead of retrying over TCP.
There’s also a race in the kernel’s connection tracking. The A and AAAA queries leave from the same socket within microseconds. Both get NATed to a CoreDNS pod, and conntrack can drop one while it inserts the other. The client waits out the full timeout, 5 seconds by default. glibc users work around it with single-request-reopen. musl has no equivalent, so Alpine pods take the full wait. The conntrack collector counts these as insert_failed.
The error depends on the runtime. These come from apps on Alpine:
Error: getaddrinfo EAI_AGAIN api.stripe.com
socket.gaierror: [Errno -3] Try again
curl: (6) Could not resolve host: api.stripe.com
The first is Node.js and the second is Python. musl’s text for EAI_AGAIN is Try again. The same Python error on a glibc image reads [Errno -3] Temporary failure in name resolution. A name that doesn’t exist reads Name does not resolve on musl, and Node.js reports it as getaddrinfo ENOTFOUND.
Go is different. A static Go binary uses Go’s own resolver by default, not musl, so Go services on Alpine dodge most of the search-list problem. Node.js, Python, Ruby, PHP and the JVM all call the C library.
In production that means slow first requests, 5-second stalls and failed calls to payment or email APIs, clustered in the busiest minute of the day.
The fix for Alpine DNS and Kubernetes ndots
Four changes, cheapest first:
- Lower
ndotson the pods that call outside services. - Write outside names as full names with a trailing dot, where your client allows it.
- Run NodeLocal DNSCache.
- Move to Alpine 3.18 or later, or to a glibc base image.
Lower ndots. Set it per pod with dnsConfig. Kubernetes merges it with the generated file:
spec:
dnsConfig:
options:
- name: ndots
value: "2"
With ndots:2, api.stripe.com goes out as a full name on the first query. Short names like redis and redis.cache still have fewer than two dots, so the search list still finds them. One catch is musl-specific. A name with at least ndots dots never falls back to the search list on musl, so redis.cache.svc breaks under ndots:2. Use redis.cache or the full redis.cache.svc.cluster.local in your config. I set ndots:2 on every Alpine deployment that calls outside APIs and grep its config for three-part service names first.
Use full names. A trailing dot tells the resolver the name is complete, so api.stripe.com. skips the search list whatever ndots says. musl handles this. Some HTTP clients pass the trailing dot into the Host header or TLS SNI, and the remote server rejects it. Test each client before you rely on it. Lowering ndots is safer when you can’t.
Run NodeLocal DNSCache. It runs a DNS cache on each node and points pods at a link-local address. Pod queries skip the iptables DNAT and conntrack, which removes the race above. The cache can also talk to CoreDNS over TCP, so a dropped UDP packet stops costing a full timeout. Install it from the Kubernetes docs, or turn it on as a GKE cluster setting. On EKS you install it yourself. CoreDNS’s autopath plugin is another route: CoreDNS answers the first search-list query with the final name, so the client stops walking the list. It needs pods verified, which costs CoreDNS memory on a large cluster.
Upgrade or switch the base image. Alpine 3.18 and later fix the TCP gap. They don’t change the search-list behavior or the missing single-request-reopen. For a service that calls many outside hosts, I move it to a Debian slim or distroless image instead of tuning around musl. A larger image costs you less than a 5-second stall at peak.
Tune timeout and attempts with care. options timeout:2 attempts:3 shortens the wait for one lost packet. It also triples the queries during a real CoreDNS outage. Lower the timeout only after the changes above, and watch CoreDNS load when you do.
Scale CoreDNS for the query rate, not the pod count. Search-list traffic multiplies queries per lookup, and every pod lookup at peak lands on CoreDNS. Raise its replicas, or run the cluster-proportional autoscaler, before you lower any other limit. The cache plugin in the default Corefile already caches answers.
On managed Kubernetes, CoreDNS runs as a provider add-on. EKS, AKS and GKE all accept dnsConfig on your pods. The provider may reset CoreDNS’s Corefile on upgrade, so keep changes to it in the add-on’s configuration rather than in a hand-edited ConfigMap.
Distro differences. The libc inside the image decides the behavior, not the node’s distro:
| Distro | C library | Who writes /etc/resolv.conf on a host | Options the resolver reads | DNS tools package |
|---|---|---|---|---|
| Ubuntu | glibc | systemd-resolved, with a stub at 127.0.0.53 | All glibc options, including single-request-reopen | bind9-dnsutils |
| Debian | glibc | ifupdown or your DHCP client | All glibc options | bind9-dnsutils |
| RHEL, Rocky, AlmaLinux | glibc | NetworkManager | All glibc options | bind-utils |
| Amazon Linux 2023 | glibc | systemd-resolved, with the stub listener turned off, so the file lists the VPC resolver | All glibc options | bind-utils |
| Alpine | musl | Your DHCP client, or the container runtime in a container | ndots, timeout and attempts only | bind-tools, or BusyBox nslookup |
In a container, the runtime writes the file. Kubernetes writes the pod file shown above, and Docker on a user-defined network points at its embedded DNS server at 127.0.0.11.
Test it on your system
Run this in a staging cluster with the same CoreDNS setup as production. A load test on your app that makes its real outside calls is the best test. To isolate DNS, run a resolver load from a pod built on your app’s base image. This script calls getaddrinfo at a fixed rate, so a slow resolver can’t slow down the request rate:
# dns_load.py sends fixed-rate getaddrinfo calls through the image's C library
import socket, sys, threading, time
from concurrent.futures import ThreadPoolExecutor
name, rate, seconds = sys.argv[1], int(sys.argv[2]), int(sys.argv[3])
lat, errors, lock = [], {}, threading.Lock()
def lookup(scheduled):
try:
socket.getaddrinfo(name, 443, proto=socket.IPPROTO_TCP)
with lock:
lat.append(time.monotonic() - scheduled)
except socket.gaierror as e:
with lock:
errors[str(e)] = errors.get(str(e), 0) + 1
with ThreadPoolExecutor(max_workers=200) as pool:
t0 = time.monotonic()
for i in range(rate * seconds):
due = t0 + i / rate
time.sleep(max(0.0, due - time.monotonic()))
pool.submit(lookup, due)
lat.sort()
if lat:
print(f"ok={len(lat)} p50={lat[len(lat) // 2]:.3f}s p99={lat[int(len(lat) * 0.99)]:.3f}s")
print("errors:", errors)
It times each lookup from its scheduled start, so time spent queued behind a slow resolver counts too. Run it at your peak lookup rate, first on the current settings, then after each change:
kubectl run dnsload --image=python:3.13-alpine --restart=Never -- sleep 3600
kubectl cp dns_load.py dnsload:/tmp/dns_load.py
kubectl exec dnsload -- python3 /tmp/dns_load.py api.stripe.com 200 300
The arguments are the name, lookups per second and duration in seconds. Swap in an outside host your app calls.
While it runs, watch CoreDNS and the node:
kubectl -n kube-system top pod -l k8s-app=kube-dns
conntrack -S | grep -o 'insert_failed=[0-9]*' # on the node
kubectl exec dnsload -- cat /etc/resolv.conf
Set targets before you begin. Your target might be p99 lookup time well under your request’s latency budget, with zero Try again errors. insert_failed should stay flat across the run. With NodeLocal DNSCache on, CoreDNS’s request rate should fall, since the node cache answers repeats.
Keep it healthy
CoreDNS exports Prometheus metrics on port 9153 by default, and node_exporter’s conntrack collector exports node_nf_conntrack_stat_insert_failed. These rules catch failing upstreams, slow answers, search-list churn and the conntrack race:
groups:
- name: cluster-dns
rules:
- alert: CoreDNSServfailRate
expr: |
sum(rate(coredns_dns_responses_total{rcode="SERVFAIL"}[5m]))
/ sum(rate(coredns_dns_responses_total[5m])) > 0.01
for: 5m
labels:
severity: critical
annotations:
summary: "CoreDNS is returning SERVFAIL for over 1% of queries"
description: "musl clients fail the whole lookup on one SERVFAIL. Check the forward plugin's upstream health with coredns_proxy_healthcheck_failures_total and the CoreDNS logs."
- alert: CoreDNSSlowAnswers
expr: |
histogram_quantile(0.99, sum by (le) (rate(coredns_dns_request_duration_seconds_bucket[5m]))) > 0.1
for: 10m
labels:
severity: warning
annotations:
summary: "CoreDNS p99 answer time is over 100 ms"
description: "Check CoreDNS CPU with kubectl top, scale its replicas, and look for a slow upstream in coredns_proxy_request_duration_seconds."
- alert: DNSSearchListChurn
expr: |
sum(rate(coredns_dns_responses_total{rcode="NXDOMAIN"}[15m]))
/ sum(rate(coredns_dns_responses_total[15m])) > 0.6
for: 30m
labels:
severity: warning
annotations:
summary: "Most CoreDNS answers are NXDOMAIN"
description: "Pods are walking the ndots:5 search list for outside names. Set dnsConfig ndots to 2 on the busiest deployments, or use full names."
- alert: ConntrackInsertFailures
expr: rate(node_nf_conntrack_stat_insert_failed[5m]) > 0
for: 10m
labels:
severity: warning
annotations:
summary: "{{ $labels.instance }} is failing conntrack inserts"
description: "Parallel UDP DNS queries are racing in conntrack, and musl clients wait out a full timeout. Run NodeLocal DNSCache on this node pool."
The thresholds are placeholders. Read a week of your own CoreDNS metrics first. A cluster full of glibc pods at ndots:5 can sit above 60 percent NXDOMAIN with no errors. Set that rule a little above your normal level.
Roll back
Save what you change first:
kubectl get deploy app -o jsonpath='{.spec.template.spec.dnsConfig}' > dnsconfig-before.json
kubectl -n kube-system get configmap coredns -o yaml > coredns-before.yaml
Delete the test pod with kubectl delete pod dnsload when you’re done. To undo ndots, remove the dnsConfig block and roll out the deployment. Pods pick up the new resolv.conf when they restart. To remove NodeLocal DNSCache, delete its DaemonSet and restore the kubelet’s clusterDNS setting if you changed it. Restore CoreDNS with kubectl apply -f coredns-before.yaml.
Roll back within the hour if names stop resolving after the change. The likely cause is a three-part service name under ndots:2. Revert anything else in a low-traffic window, since a rollout restarts every pod.
More recipes
- Out of Memory Killed Process on Linux, Fix the OOM Killer
Cap each service's memory, add a little swap and set OOM scores, so the Linux OOM killer stops the process you can lose instead of your app.
- Tuned Profiles on RHEL, and When They Override Sysctls
Check which tuned profile runs, keep sysctls in /etc/sysctl.d and carry the rest in a custom profile, so a reboot or tuned restart can't undo your tuning.
- Possible SYN Flooding on Port, Monitor the SYN Backlog
Alert on SYN cookie and listen queue counters and size the listen backlog, not tcp_max_syn_backlog, so peak traffic stops tripping SYN flood warnings.
- Systemd LimitNOFILE, Fix Open File Limits for Services
limits.conf isn't applied to a systemd service, so set LimitNOFILE in a drop-in, restart the unit and read the new limit back from /proc.