A goroutine leak is a goroutine that blocks forever and never exits. Each one keeps its stack and everything it references alive. One leaked goroutine per request at peak adds up to memory that climbs for days, until the pod gets OOMKilled.
What you miss today
Default dashboards show memory and CPU. A goroutine leak looks like slow memory growth with no change in traffic, and a deploy hides it by restarting every pod. If you deploy daily, the leak resets before it hurts. Then a holiday freeze stops the deploys, and the leak runs to the container limit.
The first report you get is often something else:
- Latency that creeps up as the GC scans a larger heap and more stacks.
OOMKilledrestarts with exit code 137 on pods that ran for days.too many open fileswhen the leaked goroutines each hold a connection.
The heap profile doesn’t point at the cause. It shows where memory was allocated, while the goroutine that keeps it alive sits blocked somewhere else. You need a goroutine count over time and the stacks of the goroutines that pile up.
The leaks I find most often take three shapes:
- A send nobody receives. A function starts workers that send results on an unbuffered channel, then returns early on the first error. The remaining workers block on
ch <-forever. - A receive nobody ends. A goroutine ranges over a channel that no code closes, or waits on a context that no code cancels.
- An HTTP response body left open. Each open client connection keeps two transport goroutines,
net/http.(*persistConn).readLoopandwriteLoop. The HTTP client connection reuse recipe fixes those.
The golang pprof goroutine metrics and profiles
Use two tools. Prometheus watches the count, and pprof shows the stacks when the count goes wrong.
Metrics. The Prometheus Go client, client_golang, registers the Go and process collectors by default. Serve them with promhttp.Handler(). Four metrics matter here:
| Metric | What it tells you |
|---|---|
go_goroutines | Goroutines alive now. A floor that rises over hours is the leak signal. |
go_memstats_heap_inuse_bytes | Heap in use. Grows with a leak that holds data. |
process_resident_memory_bytes | Resident memory of the process, stacks included. |
process_open_fds | Open file descriptors, on Linux. Rises with leaked connections. |
pprof. Importing net/http/pprof registers handlers under /debug/pprof/ on http.DefaultServeMux. Never serve that mux on a public port. I run pprof on localhost in every Go service and reach it with kubectl port-forward, so it’s there on the day I need it:
package main
import (
"log"
"net/http"
_ "net/http/pprof" // registers /debug/pprof/* on http.DefaultServeMux
"github.com/prometheus/client_golang/prometheus/promhttp"
)
func main() {
// pprof on localhost only: reach it with kubectl port-forward.
go func() {
log.Println(http.ListenAndServe("127.0.0.1:6060", http.DefaultServeMux))
}()
// Metrics on the pod IP for Prometheus to scrape.
metrics := http.NewServeMux()
metrics.Handle("GET /metrics", promhttp.Handler())
go func() {
log.Println(http.ListenAndServe(":9090", metrics))
}()
app := http.NewServeMux() // your app's own mux, without pprof
app.HandleFunc("GET /healthz", func(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(http.StatusNoContent)
})
log.Fatal(http.ListenAndServe(":8080", app))
}
kubectl port-forward reaches a port bound to the pod’s loopback address, so nothing outside the pod can call pprof.
Gin and Echo. Keep the same split. gin-contrib/pprof mounts pprof on your Gin router, which puts it on the public port unless you add auth. Echo has no pprof middleware of its own. In both, serve your router on the app port and http.DefaultServeMux on localhost as above. Leaked goroutines from a Gin or Echo handler show your handler’s function name in the stack, with framework frames such as github.com/gin-gonic/gin.(*Context).Next below it.
Reading the goroutine profile. Port-forward, then group goroutines by stack:
kubectl port-forward deploy/api 6060:6060 &
curl -s 'http://127.0.0.1:6060/debug/pprof/goroutine?debug=1' | head -40
debug=1 prints each distinct stack once, prefixed by how many goroutines share it. A leak shows up as one stack with a count far above the rest. debug=2 prints every goroutine with its state and wait time, such as [chan send, 94 minutes].
To see what grew, take two binary profiles some time apart and diff them:
curl -s -o g1.pb.gz 'http://127.0.0.1:6060/debug/pprof/goroutine'
sleep 1800
curl -s -o g2.pb.gz 'http://127.0.0.1:6060/debug/pprof/goroutine'
go tool pprof -top -base g1.pb.gz g2.pb.gz
The top lines are the stacks that gained goroutines. Run go tool pprof -http=:8081 -base g1.pb.gz g2.pb.gz for a flame graph of the same diff.
The goroutineleak profile. Go 1.27 ships a goroutineleak profile, served at /debug/pprof/goroutineleak. Go 1.26 had it as an experiment, available only in binaries built with GOEXPERIMENT=goroutineleakprofile. The runtime finds goroutines blocked on a channel, mutex or condition variable that no runnable goroutine can reach. Those can never wake up.
curl -s 'http://127.0.0.1:6060/debug/pprof/goroutineleak?debug=1' | head -20
Each request runs a garbage collection with leak detection, then lists the leaked stacks. With debug=2, the state of each leaked goroutine carries a (leaked) mark, such as [chan send (leaked)]. The detection has a limit. It misses goroutines blocked on primitives that a global variable or a running goroutine can still reach. An empty leak profile doesn’t clear you, so I start from the diff above and use the leak profile to confirm.
The alert rules
These rules use client_golang metrics plus kube-state-metrics, and assume your scrape config adds namespace and pod labels. Replace api with your container name.
groups:
- name: go-goroutine-leaks
rules:
- alert: GoGoroutineFloorRising
expr: |
min_over_time(go_goroutines[1h])
> 1.5 * max_over_time(go_goroutines[1h] offset 6h)
for: 1h
labels:
severity: warning
annotations:
summary: "Goroutines on {{ $labels.pod }} keep rising and never fall back"
description: "The lowest count in the last hour beats the highest count from six hours earlier. Port-forward to 6060 and diff two goroutine profiles 30 minutes apart with go tool pprof -base."
- alert: GoGoroutinesAboveBaseline
expr: max by (namespace, pod) (go_goroutines) > 20000
for: 15m
labels:
severity: warning
annotations:
summary: "{{ $labels.pod }} runs {{ $value }} goroutines, above its baseline"
description: "Compare with open connections in process_open_fds. If connections are flat, read /debug/pprof/goroutine?debug=1 and find the stack with the largest count."
- alert: GoMemoryGrowthToLimit
expr: |
max by (namespace, pod) (predict_linear(process_resident_memory_bytes[6h], 24 * 3600))
> on (namespace, pod)
max by (namespace, pod) (kube_pod_container_resource_limits{resource="memory", container="api"})
for: 1h
labels:
severity: warning
annotations:
summary: "{{ $labels.pod }} will reach its memory limit within a day"
description: "Memory has grown steadily for six hours. Check go_goroutines on the same pod; if it rose too, take a goroutine profile before the pod restarts and loses the evidence."
- alert: GoLeakEndedInOOMKill
expr: |
increase(kube_pod_container_status_restarts_total{container="api"}[10m]) > 0
and on (namespace, pod, container)
kube_pod_container_status_last_terminated_reason{reason="OOMKilled", container="api"} == 1
for: 0m
labels:
severity: critical
annotations:
summary: "Go container {{ $labels.pod }} was OOM-killed"
description: "Look at go_goroutines and memory on this pod over the last day. A slow climb before the kill means a leak; take goroutine profiles from a sibling pod now."
The first rule carries most of the weight. Comparing the lowest point now with the highest point six hours ago ignores traffic spikes. A busy hour raises the count and a quiet one lowers it, but a leak lifts the floor itself. The rule needs pods older than seven hours, so it stays silent after each deploy.
Test the alerts
Build a staging-only image with a handler that leaks on purpose. A build tag keeps it out of production builds:
//go:build leaktest
package main
import (
"net/http"
"strconv"
)
// Staging-only handler: each call strands n goroutines on a channel
// that nothing reads. Build with -tags leaktest.
func init() {
http.HandleFunc("POST /debug/leak", func(w http.ResponseWriter, r *http.Request) {
n, _ := strconv.Atoi(r.URL.Query().Get("n"))
done := make(chan struct{})
for range n {
go func() { done <- struct{}{} }()
}
w.WriteHeader(http.StatusAccepted)
})
}
The handler registers on http.DefaultServeMux, so it only answers on the localhost pprof port. Build with go build -tags leaktest, deploy it as one staging pod and leak a steady trickle through a port-forward:
kubectl port-forward deploy/api-leaktest 6060:6060 &
while true; do curl -s -X POST 'http://127.0.0.1:6060/debug/leak?n=200'; sleep 60; done
Then confirm each step:
go_goroutinesclimbs in a straight line on that pod.GoGoroutineFloorRisingfires about eight hours after the pod starts: seven hours of history, then theforhour.- The profile diff from the last section names
main.init.0.func1.1, the leaking closure. - On Go 1.27,
/debug/pprof/goroutineleak?debug=1lists the same stack. - Stop the loop and delete the pod. The alerts clear once the new pod’s metrics replace the old series.
To test GoGoroutinesAboveBaseline without waiting, raise n until the count passes your threshold. Run your normal k6 load test at the same time, so you also see how the leak shifts latency and memory under traffic.
In CI, goleak catches leaks before deploy. Add goleak.VerifyTestMain(m) to a package’s TestMain, and the test run fails if goroutines outlive the tests.
Tune the noise
The goroutine count tracks open connections as well as leaks. net/http runs one goroutine per server connection, so idle keep-alive clients raise the count. A pod serving WebSockets or long polls can run tens of thousands of goroutines in normal operation. Set the thresholds from your own history:
quantile_over_time(0.99, max(go_goroutines{namespace="prod", pod=~"api-.*"})[7d:5m])
Set GoGoroutinesAboveBaseline a margin above that number, per service, not one figure for every service. The 20000 in the rule is a placeholder.
For the floor rule, the 1.5 ratio and the 6-hour offset suit a service with a daily traffic cycle. If your traffic ramps up over a whole morning, the floor rises for real. Use a ratio of 2 there, or compare with the same hour a day earlier using offset 1d. Pods that restart often never build six hours of history. For those, the memory rule and the OOM rule cover you.
Keep for long on the leak rules. A leak takes hours to hurt, so an alert that waits an hour costs nothing and skips the noise from deploys and traffic bursts. The OOM rule has for: 0m because the kill already happened.
More recipes
- GOMAXPROCS and GOMEMLIMIT for Go on Kubernetes
Match GOMAXPROCS to the pod's CPU limit and set GOMEMLIMIT just under its memory limit, so a Go service stops getting throttled and OOM-killed at peak.
- Go Too Many Open Files and HTTP Connection Reuse
Share one Go HTTP client per service and set server timeouts, so peak traffic stops leaking file descriptors and your service keeps accepting connections.
- Improve Cache Hit Ratio to Protect Your Origin at Peak
Trim the cache key, send cache headers from the origin and serve stale while revalidating, so the CDN absorbs peak traffic instead of passing it to your origin.
- Fix ALB 502 Bad Gateway and 504 Gateway Timeout
Set each target's keep-alive above the ALB idle timeout and keep slow requests under it, so the load balancer stops answering peak traffic with 502 and 504.