GOMAXPROCS and GOMEMLIMIT for Go on Kubernetes

Match GOMAXPROCS to the pod's CPU limit and set GOMEMLIMIT just under its memory limit, so a Go service stops getting throttled and OOM-killed at peak.

Published

Applies to Go 1.19 and later for GOMEMLIMIT; container-aware GOMAXPROCS from Go 1.25, when go.mod says go 1.25 or later

GOMAXPROCS on Kubernetes decides how many threads run Go code at once, and before Go 1.25 it ignored the pod’s CPU limit. GOMEMLIMIT tells the garbage collector how much memory the container has. Without it, the Go runtime has no idea a limit exists. Neither setting logs anything when it’s wrong, so you find out at the next peak.

What breaks at peak

CPU throttling. Before Go 1.25, GOMAXPROCS defaulted to runtime.NumCPU(), the cores the process can see. A pod with a 2-CPU limit on a 32-core node ran 32 threads of Go code. The kernel’s CFS quota hands that pod 2 CPUs of time per 100 ms period. Thirty-two busy threads burn the quota early in the period, then the whole pod waits for the next one.

On your dashboards this looks like p99 spikes with CPU usage under the limit, because usage averages over the period. container_cpu_cfs_throttled_periods_total climbs. The garbage collector makes it worse. It aims to use a quarter of GOMAXPROCS for background marking, so a GC cycle sized for 32 cores lands on a 2-CPU budget. The Kubernetes CPU throttling recipe covers the limit side.

OOM kills. The Go GC paces itself with GOGC, 100 by default. The heap may grow to about twice the live heap before the next cycle. At peak, the live heap grows with in-flight requests, and the next target can pass the container limit. The kernel kills the process with exit code 137, Kubernetes reports OOMKilled, and every in-flight request on that pod fails. If the pod also restarts in a loop, see the OOMKilled and exit code 137 recipe.

The GOMAXPROCS fix

Go 1.25 made the default container-aware on Linux. At startup the runtime reads the cgroup CPU bandwidth limit, which is the Kubernetes CPU limit. If the limit is lower than the CPU count, GOMAXPROCS uses the limit. The runtime also rechecks it while running, so a resized pod picks up the new value.

Four details decide what your pod gets:

  • Your go.mod line turns it on. The behavior sits behind the containermaxprocs and updatemaxprocs GODEBUG settings. Their defaults follow the go line in the main module’s go.mod, not the toolchain version. A binary built with Go 1.27 from a module that says go 1.22 keeps the old behavior.
  • It rounds up, with a floor of 2. A 1500m limit gives 2. A 500m limit gives 2 as well, unless the node has only one CPU.
  • Requests don’t count. With a CPU request and no limit, there’s no quota to read, and GOMAXPROCS equals the node’s cores.
  • Setting it by hand turns it off. A GOMAXPROCS environment variable or a call to runtime.GOMAXPROCS disables both the cgroup read and the updates.

Check go.mod first:

module example.com/api

go 1.25

If you can’t raise the go line yet, opt in from your main package with a //go:debug directive above the package clause:

//go:debug containermaxprocs=1
//go:debug updatemaxprocs=1
package main

On Go 1.24 and older, go.uber.org/automaxprocs sets GOMAXPROCS from the quota at startup. It rounds down with a floor of 1, so a 1500m limit gives 1. Remove it once you’re on Go 1.25 with the new default. It calls runtime.GOMAXPROCS, which switches off the runtime’s own updates.

Pods without a CPU limit. Many teams drop CPU limits to avoid throttling. The Go runtime then sees every core on the node. I set GOMAXPROCS from the CPU request in that case, so the GC and scheduler size themselves to the CPU the pod can count on. The downward API rounds a CPU value up to a whole core:

# Deployment, changed lines only
env:
  - name: GOMAXPROCS
    valueFrom:
      resourceFieldRef:
        resource: requests.cpu

You give up the runtime’s updates here, which matters only if you resize pods in place.

The GOMEMLIMIT fix

GOMEMLIMIT sets a soft limit on the memory the Go runtime manages. Near the limit, the GC runs more often to stay under it. The Go GC guide’s rule of thumb is to leave 5 to 10% of the container limit as headroom for memory the runtime doesn’t track. Memory from cgo libraries and memory-mapped files falls outside it, so leave more if you use them.

Set it in the pod spec next to the limit:

# Deployment, changed lines only
resources:
  requests:
    cpu: "2"
    memory: 1Gi
  limits:
    cpu: "2"
    memory: 1Gi
env:
  - name: GOMEMLIMIT
    value: "920MiB"

The value takes IEC suffixes: B, KiB, MiB, GiB and TiB. Kubernetes quantities such as 920Mi aren’t valid, and the runtime refuses to start with this error:

GOMEMLIMIT=920Mi
fatal error: malformed GOMEMLIMIT; see `go doc runtime/debug.SetMemoryLimit`

A literal in the manifest drifts when someone changes the memory limit and forgets the env var. To keep the two in step, read the cgroup limit at startup and set 90% of it from code:

package main

import (
	"log"
	"os"
	"runtime"
	"runtime/debug"
	"strconv"
	"strings"
)

// setMemoryLimit sets the Go soft memory limit to ratio times the cgroup v2
// memory limit. It leaves an explicit GOMEMLIMIT alone.
func setMemoryLimit(ratio float64) {
	if os.Getenv("GOMEMLIMIT") != "" {
		return
	}
	raw, err := os.ReadFile("/sys/fs/cgroup/memory.max")
	if err != nil {
		return // not on cgroup v2
	}
	value := strings.TrimSpace(string(raw))
	if value == "max" {
		return // no container limit
	}
	limit, err := strconv.ParseInt(value, 10, 64)
	if err != nil {
		return
	}
	debug.SetMemoryLimit(int64(float64(limit) * ratio))
}

func main() {
	setMemoryLimit(0.9)
	log.Printf("GOMAXPROCS=%d GOMEMLIMIT=%d", runtime.GOMAXPROCS(0), debug.SetMemoryLimit(-1))
	// start your server here
}

debug.SetMemoryLimit(-1) reads the current limit without changing it. The startup log line is the first thing I check on any Go pod, because it shows both values the runtime settled on.

Leave GOGC at 100. I don’t pair GOMEMLIMIT with GOGC=off on web services. The GC then waits until the heap nears the limit. Each pod runs close to it, and a burst of in-flight requests has no room. A memory limit stops a heap target that overshoots. It can’t make a live heap fit, because the GC caps its own CPU near 50% and then lets the heap pass the limit.

Gin and Echo. Both frameworks run on net/http, and their handlers run as ordinary goroutines. The environment variables apply unchanged. If you set the limit from code, call setMemoryLimit at the top of main, before r.Run() in Gin or e.Start() in Echo.

Test it on your system

Run a staging deployment with production’s CPU and memory limits and the same Go version and go.mod line. Confirm the values first:

kubectl logs deploy/api | grep GOMAXPROCS
kubectl exec deploy/api -- cat /sys/fs/cgroup/cpu.max /sys/fs/cgroup/memory.max

cpu.max prints quota and period, such as 200000 100000 for 2 CPUs. memory.max prints bytes.

Then load the pod at a fixed arrival rate with k6, so a slow pod doesn’t slow the test:

// go-peak.js
import http from 'k6/http';

export const options = {
  scenarios: {
    peak: {
      executor: 'constant-arrival-rate',
      rate: Number(__ENV.RATE || 100),
      timeUnit: '1s',
      duration: '15m',
      preAllocatedVUs: 200,
      maxVUs: 1000,
    },
  },
  thresholds: {
    http_req_failed: ['rate<0.01'],
    http_req_duration: ['p(99)<300'],
  },
};

export default function () {
  http.get(`${__ENV.BASE_URL}/api/search?q=shoes`);
}
k6 run -e BASE_URL=https://staging.example.com -e RATE=400 go-peak.js

Set RATE to one pod’s share of peak traffic and replace the 300 ms with your own budget. Pick an endpoint that allocates the way production does, since GC load follows allocation. Then repeat the run with the old settings to see the difference on your own workload. That means GOMAXPROCS set to the node’s core count and no GOMEMLIMIT.

Watch these while it runs:

  • Throttling. The ratio of container_cpu_cfs_throttled_periods_total to container_cpu_cfs_periods_total from cAdvisor.
  • Memory. container_memory_working_set_bytes against the limit, and go_memstats_heap_inuse_bytes from the app.
  • GC. Set GODEBUG=gctrace=1 on the staging pod. Each GC prints a line to stderr, and the percentage after the @ timestamp is the share of CPU the GC has used since start.

The pass criteria are yours to set. A fair start: p99 inside your budget, a throttled-period ratio that stays low and steady, and no restarts. Memory should level off under the limit with the GC share flat. A GC share that keeps climbing near the limit means the live heap doesn’t fit. Give the pod more memory, or send it fewer requests at once.

Keep it healthy

The Prometheus Go client, client_golang 1.20 and later, exports go_sched_gomaxprocs_threads and go_gc_gomemlimit_bytes by default. cAdvisor and kube-state-metrics cover the container. These rules assume your scrape config adds namespace and pod labels to app metrics and that the container is called api:

groups:
  - name: go-runtime-limits
    rules:
      - alert: GoPodCPUThrottled
        expr: |
          sum by (namespace, pod) (rate(container_cpu_cfs_throttled_periods_total{container="api"}[5m]))
            / sum by (namespace, pod) (rate(container_cpu_cfs_periods_total{container="api"}[5m]))
            > 0.25
        for: 15m
        labels:
          severity: warning
        annotations:
          summary: "Go pod {{ $labels.pod }} is throttled in over 25% of CPU periods"
          description: "Check the GOMAXPROCS startup log line against the CPU limit. If they match, the pod needs more CPU or more replicas."
      - alert: GoMaxProcsAboveCPULimit
        expr: |
          max by (namespace, pod) (go_sched_gomaxprocs_threads)
            > on (namespace, pod)
          clamp_min(ceil(max by (namespace, pod) (kube_pod_container_resource_limits{resource="cpu", container="api"})), 2)
        for: 30m
        labels:
          severity: warning
        annotations:
          summary: "GOMAXPROCS on {{ $labels.pod }} is above the pod's CPU limit"
          description: "The runtime is not reading the cgroup limit. Raise the go line in go.mod to 1.25 or later, or remove a GOMAXPROCS env var or automaxprocs call."
      - alert: GoMemoryLimitUnset
        expr: max by (namespace, pod) (go_gc_gomemlimit_bytes) >= 9e18
        for: 1h
        labels:
          severity: warning
        annotations:
          summary: "Go pod {{ $labels.pod }} runs with no GOMEMLIMIT"
          description: "The GC doesn't know the container limit. Set GOMEMLIMIT to about 90% of the memory limit, with a MiB or GiB suffix."
      - alert: GoContainerOOMKilled
        expr: |
          increase(kube_pod_container_status_restarts_total{container="api"}[10m]) > 0
            and on (namespace, pod, container)
          kube_pod_container_status_last_terminated_reason{reason="OOMKilled", container="api"} == 1
        for: 0m
        labels:
          severity: critical
        annotations:
          summary: "Go container {{ $labels.pod }} was OOM-killed"
          description: "Check GOMEMLIMIT against the limit and go_memstats_heap_inuse_bytes before the kill. A live heap near the limit needs more memory, not a lower GOMEMLIMIT."

An unset limit reports math.MaxInt64, about 9.2e18, which the third rule matches. Set the 25% throttling threshold from your own baseline. Some throttling at peak is normal on a pod with a tight limit.

Roll back

Save the current env and resources before the change:

kubectl get deploy api -o jsonpath='{.spec.template.spec.containers[0].env}{"\n"}{.spec.template.spec.containers[0].resources}' > go-limits-before.txt

To undo the env vars, remove them and let the rollout replace the pods:

kubectl set env deploy/api GOMEMLIMIT- GOMAXPROCS-

To undo the go.mod change without a code revert, set GODEBUG=containermaxprocs=0,updatemaxprocs=0 on the container. If the new memory limit brought a GC share that eats your CPU, roll back now. For anything else, revert during your lowest-traffic hour, since the rollout restarts every pod.

More recipes

Further reading