Find a Goroutine Leak with pprof and Prometheus

Alert on a rising goroutine floor and read the golang pprof goroutine profiles, so you find a goroutine leak days before it ends in an OOM kill.

Published

Applies to Go 1.21 and later; goroutineleak profile in Go 1.27, or Go 1.26 built with GOEXPERIMENT=goroutineleakprofile; client_golang 1.20 and later

A goroutine leak is a goroutine that blocks forever and never exits. Each one keeps its stack and everything it references alive. One leaked goroutine per request at peak adds up to memory that climbs for days, until the pod gets OOMKilled.

What you miss today

Default dashboards show memory and CPU. A goroutine leak looks like slow memory growth with no change in traffic, and a deploy hides it by restarting every pod. If you deploy daily, the leak resets before it hurts. Then a holiday freeze stops the deploys, and the leak runs to the container limit.

The first report you get is often something else:

  • Latency that creeps up as the GC scans a larger heap and more stacks.
  • OOMKilled restarts with exit code 137 on pods that ran for days.
  • too many open files when the leaked goroutines each hold a connection.

The heap profile doesn’t point at the cause. It shows where memory was allocated, while the goroutine that keeps it alive sits blocked somewhere else. You need a goroutine count over time and the stacks of the goroutines that pile up.

The leaks I find most often take three shapes:

  • A send nobody receives. A function starts workers that send results on an unbuffered channel, then returns early on the first error. The remaining workers block on ch <- forever.
  • A receive nobody ends. A goroutine ranges over a channel that no code closes, or waits on a context that no code cancels.
  • An HTTP response body left open. Each open client connection keeps two transport goroutines, net/http.(*persistConn).readLoop and writeLoop. The HTTP client connection reuse recipe fixes those.

The golang pprof goroutine metrics and profiles

Use two tools. Prometheus watches the count, and pprof shows the stacks when the count goes wrong.

Metrics. The Prometheus Go client, client_golang, registers the Go and process collectors by default. Serve them with promhttp.Handler(). Four metrics matter here:

MetricWhat it tells you
go_goroutinesGoroutines alive now. A floor that rises over hours is the leak signal.
go_memstats_heap_inuse_bytesHeap in use. Grows with a leak that holds data.
process_resident_memory_bytesResident memory of the process, stacks included.
process_open_fdsOpen file descriptors, on Linux. Rises with leaked connections.

pprof. Importing net/http/pprof registers handlers under /debug/pprof/ on http.DefaultServeMux. Never serve that mux on a public port. I run pprof on localhost in every Go service and reach it with kubectl port-forward, so it’s there on the day I need it:

package main

import (
	"log"
	"net/http"
	_ "net/http/pprof" // registers /debug/pprof/* on http.DefaultServeMux

	"github.com/prometheus/client_golang/prometheus/promhttp"
)

func main() {
	// pprof on localhost only: reach it with kubectl port-forward.
	go func() {
		log.Println(http.ListenAndServe("127.0.0.1:6060", http.DefaultServeMux))
	}()

	// Metrics on the pod IP for Prometheus to scrape.
	metrics := http.NewServeMux()
	metrics.Handle("GET /metrics", promhttp.Handler())
	go func() {
		log.Println(http.ListenAndServe(":9090", metrics))
	}()

	app := http.NewServeMux() // your app's own mux, without pprof
	app.HandleFunc("GET /healthz", func(w http.ResponseWriter, r *http.Request) {
		w.WriteHeader(http.StatusNoContent)
	})
	log.Fatal(http.ListenAndServe(":8080", app))
}

kubectl port-forward reaches a port bound to the pod’s loopback address, so nothing outside the pod can call pprof.

Gin and Echo. Keep the same split. gin-contrib/pprof mounts pprof on your Gin router, which puts it on the public port unless you add auth. Echo has no pprof middleware of its own. In both, serve your router on the app port and http.DefaultServeMux on localhost as above. Leaked goroutines from a Gin or Echo handler show your handler’s function name in the stack, with framework frames such as github.com/gin-gonic/gin.(*Context).Next below it.

Reading the goroutine profile. Port-forward, then group goroutines by stack:

kubectl port-forward deploy/api 6060:6060 &
curl -s 'http://127.0.0.1:6060/debug/pprof/goroutine?debug=1' | head -40

debug=1 prints each distinct stack once, prefixed by how many goroutines share it. A leak shows up as one stack with a count far above the rest. debug=2 prints every goroutine with its state and wait time, such as [chan send, 94 minutes].

To see what grew, take two binary profiles some time apart and diff them:

curl -s -o g1.pb.gz 'http://127.0.0.1:6060/debug/pprof/goroutine'
sleep 1800
curl -s -o g2.pb.gz 'http://127.0.0.1:6060/debug/pprof/goroutine'
go tool pprof -top -base g1.pb.gz g2.pb.gz

The top lines are the stacks that gained goroutines. Run go tool pprof -http=:8081 -base g1.pb.gz g2.pb.gz for a flame graph of the same diff.

The goroutineleak profile. Go 1.27 ships a goroutineleak profile, served at /debug/pprof/goroutineleak. Go 1.26 had it as an experiment, available only in binaries built with GOEXPERIMENT=goroutineleakprofile. The runtime finds goroutines blocked on a channel, mutex or condition variable that no runnable goroutine can reach. Those can never wake up.

curl -s 'http://127.0.0.1:6060/debug/pprof/goroutineleak?debug=1' | head -20

Each request runs a garbage collection with leak detection, then lists the leaked stacks. With debug=2, the state of each leaked goroutine carries a (leaked) mark, such as [chan send (leaked)]. The detection has a limit. It misses goroutines blocked on primitives that a global variable or a running goroutine can still reach. An empty leak profile doesn’t clear you, so I start from the diff above and use the leak profile to confirm.

The alert rules

These rules use client_golang metrics plus kube-state-metrics, and assume your scrape config adds namespace and pod labels. Replace api with your container name.

groups:
  - name: go-goroutine-leaks
    rules:
      - alert: GoGoroutineFloorRising
        expr: |
          min_over_time(go_goroutines[1h])
            > 1.5 * max_over_time(go_goroutines[1h] offset 6h)
        for: 1h
        labels:
          severity: warning
        annotations:
          summary: "Goroutines on {{ $labels.pod }} keep rising and never fall back"
          description: "The lowest count in the last hour beats the highest count from six hours earlier. Port-forward to 6060 and diff two goroutine profiles 30 minutes apart with go tool pprof -base."
      - alert: GoGoroutinesAboveBaseline
        expr: max by (namespace, pod) (go_goroutines) > 20000
        for: 15m
        labels:
          severity: warning
        annotations:
          summary: "{{ $labels.pod }} runs {{ $value }} goroutines, above its baseline"
          description: "Compare with open connections in process_open_fds. If connections are flat, read /debug/pprof/goroutine?debug=1 and find the stack with the largest count."
      - alert: GoMemoryGrowthToLimit
        expr: |
          max by (namespace, pod) (predict_linear(process_resident_memory_bytes[6h], 24 * 3600))
            > on (namespace, pod)
          max by (namespace, pod) (kube_pod_container_resource_limits{resource="memory", container="api"})
        for: 1h
        labels:
          severity: warning
        annotations:
          summary: "{{ $labels.pod }} will reach its memory limit within a day"
          description: "Memory has grown steadily for six hours. Check go_goroutines on the same pod; if it rose too, take a goroutine profile before the pod restarts and loses the evidence."
      - alert: GoLeakEndedInOOMKill
        expr: |
          increase(kube_pod_container_status_restarts_total{container="api"}[10m]) > 0
            and on (namespace, pod, container)
          kube_pod_container_status_last_terminated_reason{reason="OOMKilled", container="api"} == 1
        for: 0m
        labels:
          severity: critical
        annotations:
          summary: "Go container {{ $labels.pod }} was OOM-killed"
          description: "Look at go_goroutines and memory on this pod over the last day. A slow climb before the kill means a leak; take goroutine profiles from a sibling pod now."

The first rule carries most of the weight. Comparing the lowest point now with the highest point six hours ago ignores traffic spikes. A busy hour raises the count and a quiet one lowers it, but a leak lifts the floor itself. The rule needs pods older than seven hours, so it stays silent after each deploy.

Test the alerts

Build a staging-only image with a handler that leaks on purpose. A build tag keeps it out of production builds:

//go:build leaktest

package main

import (
	"net/http"
	"strconv"
)

// Staging-only handler: each call strands n goroutines on a channel
// that nothing reads. Build with -tags leaktest.
func init() {
	http.HandleFunc("POST /debug/leak", func(w http.ResponseWriter, r *http.Request) {
		n, _ := strconv.Atoi(r.URL.Query().Get("n"))
		done := make(chan struct{})
		for range n {
			go func() { done <- struct{}{} }()
		}
		w.WriteHeader(http.StatusAccepted)
	})
}

The handler registers on http.DefaultServeMux, so it only answers on the localhost pprof port. Build with go build -tags leaktest, deploy it as one staging pod and leak a steady trickle through a port-forward:

kubectl port-forward deploy/api-leaktest 6060:6060 &
while true; do curl -s -X POST 'http://127.0.0.1:6060/debug/leak?n=200'; sleep 60; done

Then confirm each step:

  • go_goroutines climbs in a straight line on that pod. GoGoroutineFloorRising fires about eight hours after the pod starts: seven hours of history, then the for hour.
  • The profile diff from the last section names main.init.0.func1.1, the leaking closure.
  • On Go 1.27, /debug/pprof/goroutineleak?debug=1 lists the same stack.
  • Stop the loop and delete the pod. The alerts clear once the new pod’s metrics replace the old series.

To test GoGoroutinesAboveBaseline without waiting, raise n until the count passes your threshold. Run your normal k6 load test at the same time, so you also see how the leak shifts latency and memory under traffic.

In CI, goleak catches leaks before deploy. Add goleak.VerifyTestMain(m) to a package’s TestMain, and the test run fails if goroutines outlive the tests.

Tune the noise

The goroutine count tracks open connections as well as leaks. net/http runs one goroutine per server connection, so idle keep-alive clients raise the count. A pod serving WebSockets or long polls can run tens of thousands of goroutines in normal operation. Set the thresholds from your own history:

quantile_over_time(0.99, max(go_goroutines{namespace="prod", pod=~"api-.*"})[7d:5m])

Set GoGoroutinesAboveBaseline a margin above that number, per service, not one figure for every service. The 20000 in the rule is a placeholder.

For the floor rule, the 1.5 ratio and the 6-hour offset suit a service with a daily traffic cycle. If your traffic ramps up over a whole morning, the floor rises for real. Use a ratio of 2 there, or compare with the same hour a day earlier using offset 1d. Pods that restart often never build six hours of history. For those, the memory rule and the OOM rule cover you.

Keep for long on the leak rules. A leak takes hours to hurt, so an alert that waits an hour costs nothing and skips the noise from deploys and traffic bursts. The OOM rule has for: 0m because the kill already happened.

More recipes

Further reading