G1 vs ZGC is the wrong first question when p99 spikes. Start by measuring how long your JVM pauses, how often, and why. Those numbers tell you whether to fix the heap, the CPU limit or the collector.
What you miss today
Most JVM dashboards show heap used and an average GC time. An average hides the pause that matters: one stop-the-world collection that freezes every thread past your latency budget. During that pause, every request in the pod waits, whatever endpoint it hit.
That gives GC pauses a fingerprint you can spot without GC metrics:
- p99 jumps on every endpoint of one pod at the same moment, while CPU and traffic look normal.
- Liveness probes time out in bursts, and the kubelet restarts a pod that wasn’t broken.
- HikariCP logs
Thread starvation or clock leap detected (housekeeper delta=...), because its housekeeping thread froze along with everything else. - The GC log shows
Pause Fulllines, which on G1 means the concurrent collector fell behind.
Two container settings cause pauses that no collector setting fixes.
Small pods may not run G1 at all. Before JDK 27, the JVM picks the Serial collector when it sees fewer than 2 CPUs or less than 1792 MB of memory. A 1-CPU pod then runs every collection on one thread, with all request threads stopped. JDK 27 made G1 the default everywhere (JEP 523). Check what your pods run:
kubectl exec deploy/api -- java -XX:+PrintFlagsFinal -version \
| grep -E ' Use(Serial|Parallel|G1|Z|Shenandoah)GC '
CPU throttling stretches every pause. G1 runs its pause work on several threads. Under a tight CPU limit, the kernel throttles them mid-pause, and a short pause turns long. The CPU throttling recipe shows how to spot that.
G1 or ZGC. G1 has been the default since JDK 9. It aims for -XX:MaxGCPauseMillis, 200 ms by default, and it can miss that target under load. ZGC does its marking and relocation while your threads run, so its pauses don’t grow with the heap. It pays for that with more CPU and a need for more free heap. When ZGC can’t keep up with allocation, threads stall and the log shows Allocation Stall.
The exporter and metrics
Spring Boot Actuator exposes Micrometer’s JVM metrics at /actuator/prometheus once micrometer-registry-prometheus is on the classpath. Outside Spring, bind Micrometer’s JvmGcMetrics yourself. The Prometheus JMX exporter agent also works, but it uses other series names, so adjust the rules below.
Changed lines only, in application.properties. The histogram setting gives you GC pause percentiles instead of a sum and a max:
management.endpoints.web.exposure.include=health,prometheus
management.metrics.distribution.percentiles-histogram.jvm.gc.pause=true
management.metrics.distribution.percentiles-histogram.http.server.requests=true
Add a GC log for the detail metrics can’t carry, such as the phase that took the time. In a container, stdout works well:
-Xlog:gc*:stdout:time,uptime,level,tags
These series answer the questions that matter:
| Metric | What it tells you |
|---|---|
jvm_gc_pause_seconds_count, _sum | How many pauses, and total time paused. rate() of the sum is the share of wall time the JVM spent stopped. |
jvm_gc_pause_seconds_max | The longest pause in the last step window. A spike here lines up with a p99 spike. |
jvm_gc_pause_seconds_bucket | Pause percentiles, once the histogram setting is on. |
jvm_gc_concurrent_phase_time_seconds | Time in concurrent phases, such as ZGC and Shenandoah cycles. Not a pause, but it uses CPU your requests want. |
jvm_gc_live_data_size_bytes, jvm_gc_max_data_size_bytes | Live data after a collection against the old generation’s maximum. A ratio that climbs ends in full GCs. |
jvm_gc_memory_allocated_bytes_total | Allocation rate. A jump after a deploy explains more GC with no traffic change. |
process_uptime_seconds | Lets you mute alerts during warm-up. |
Each pause series carries three labels. gc names the collector, such as G1 Young Generation or G1 Old Generation. On G1, G1 Old Generation counts full collections. On generational ZGC, pauses report under names that end in Pauses. action says minor or major, and cause says why, such as G1 Evacuation Pause or System.gc().
The alert rules
These rules cover long pauses, a high share of time paused and full collections. The thresholds are starting points. The last section shows how to set yours.
groups:
- name: jvm-gc
rules:
- alert: JvmGcPauseLong
expr: |
max by (job, instance) (jvm_gc_pause_seconds_max) > 0.5
and on (job, instance) max by (job, instance) (process_uptime_seconds) > 300
for: 2m
labels:
severity: warning
annotations:
summary: "GC pauses over 500 ms on {{ $labels.instance }}"
description: "Every request in this JVM waits during the pause. Read the GC log for the pause cause, then check heap size and CPU throttling before you change collector."
- alert: JvmGcPauseP99High
expr: |
histogram_quantile(0.99,
sum by (le, job) (rate(jvm_gc_pause_seconds_bucket[10m]))
) > 0.2
for: 15m
labels:
severity: warning
annotations:
summary: "p99 GC pause for {{ $labels.job }} is above 200 ms"
description: "GC is taking a large share of your latency budget. Compare live data with the heap maximum, then try a lower MaxGCPauseMillis or ZGC in staging."
- alert: JvmGcTimeHigh
expr: |
sum by (job, instance) (rate(jvm_gc_pause_seconds_sum[5m])) > 0.1
and on (job, instance) max by (job, instance) (process_uptime_seconds) > 300
for: 10m
labels:
severity: critical
annotations:
summary: "{{ $labels.instance }} spends over 10% of its time in GC pauses"
description: "The heap is too small for the live data or allocation has jumped. Check jvm_gc_live_data_size_bytes, and take a heap dump with jcmd 1 GC.heap_dump if it keeps climbing."
- alert: JvmFullGc
expr: |
sum by (job, instance) (increase(jvm_gc_pause_seconds_count{gc="G1 Old Generation"}[10m])) > 0
for: 1m
labels:
severity: warning
annotations:
summary: "Full GC on {{ $labels.instance }}"
description: "G1 fell back to a stop-the-world full collection. Check the cause label; for anything but System.gc(), raise the heap or find the leak with the heap recipe."
JvmFullGc uses G1’s collector name. On Parallel or Serial, match action="end of major GC" instead.
A full GC often follows a heap that’s too small for its live data. The JVM heap in containers recipe sizes the heap from the container limit, and it alerts on live data near the maximum. Elasticsearch runs on the JVM too, and its heap limits show up as circuit_breaking_exception, which the Elasticsearch rejections recipe covers.
Test the alerts
Load the rules into a staging Prometheus that scrapes a staging pod, then cause each failure on purpose.
Full GC. Ask the JVM for one. On G1, System.gc() runs a full collection unless someone set -XX:+ExplicitGCInvokesConcurrent:
kubectl exec deploy/api -- jcmd 1 GC.run
JvmFullGc should go pending, then fire a minute later with cause="System.gc()" on the series. It clears once the 10-minute window passes with no new full GC.
Long pauses and GC time. Shrink the heap in staging until the live data almost fills it. Set -XX:MaxRAMPercentage low enough that the heap runs near full, then hold load with a fixed arrival rate:
import http from 'k6/http';
export const options = {
scenarios: {
gc: {
executor: 'constant-arrival-rate',
rate: 200,
timeUnit: '1s',
duration: '20m',
preAllocatedVUs: 200,
maxVUs: 1000,
},
},
};
export default function () {
http.get(`${__ENV.BASE_URL}/api/reports/daily`);
}
Pick an endpoint that allocates a lot, such as a report or a large JSON response. Set rate near your peak for one pod. A closed-loop tool would slow down as the pauses grow and hide them, so keep the arrival rate fixed.
While it runs, check the alert state and the raw series:
kubectl logs -f deploy/api | grep -E 'Pause (Young|Full)'
curl -s 'http://prometheus.staging:9090/api/v1/query' \
--data-urlencode 'query=ALERTS{alertname=~"JvmGc.*|JvmFullGc"}'
Each alert passes the test when it fires while the problem is real and clears within its window after you stop. Restore the heap setting afterwards. If an alert never fires, query its expression in Prometheus and check the label values your pods report.
Tune the noise
Set thresholds from your own baseline, not from these defaults. Pull a week of pause data at your busiest hours:
max_over_time(max by (job) (jvm_gc_pause_seconds_max)[7d:5m])
histogram_quantile(0.99, sum by (le, job) (rate(jvm_gc_pause_seconds_bucket[1h])))
sum by (job, instance) (rate(jvm_gc_pause_seconds_sum[1h]))
Then tie the pause thresholds to your request latency budget. If your p99 budget is 500 ms and normal work takes 300 ms of it, GC can have the rest. Set JvmGcPauseLong near that leftover, and JvmGcPauseP99High below it.
A few more adjustments cut false pages:
- Warm-up. Startup triggers collections while caches fill and classes load. The
process_uptime_seconds > 300guard mutes the first five minutes. Stretch it if your service warms up longer. - Deliberate
System.gc(). Some libraries and RMI call it on a schedule. Addcause!="System.gc()"toJvmFullGc, or run with-XX:+ExplicitGCInvokesConcurrent. fordurations. One long pause is a single sample, soJvmGcPauseLongwaits 2 minutes to catch repeats. Time-share alerts wait longer, because a slow climb matters more than one bad minute.
When the data says switch. First rule out the cheap fixes. ZGC won’t cure a heap too small for its live data. It won’t fix a 1-CPU pod on the Serial collector or CPU throttling either. If G1’s young and mixed pauses still break your budget after that, try a lower target first:
-XX:MaxGCPauseMillis=100
G1 meets a lower target with a smaller young generation and more frequent collections, at some cost in throughput. If that still misses, try ZGC in staging:
-XX:+UseZGC
On JDK 23 and later that’s generational ZGC. On JDK 21 and 22, add -XX:+ZGenerational, since the older mode is the default there. Give ZGC more heap than G1 needed, because it collects while your code allocates.
Run the same fixed-rate test against both and compare p99 latency, CPU use, jvm_gc_pause_seconds_max and any Allocation Stall lines. I move to ZGC when G1 still breaks the budget after the heap fits and the pod has CPU to spare. On a one or two CPU pod, I fix the CPU limit first.
Roll back
The alert rules are additive. Delete the rule group from your Prometheus config to remove them.
A collector switch is a JVM flag. Save the current flags with kubectl get deployment api -o yaml > api-before.yaml before you change them. To revert, remove -XX:+UseZGC or the new pause target and redeploy, or run kubectl rollout undo deployment/api.
If ZGC brings allocation stalls, throttling or OOMKilled pods at peak, roll back now. If it only failed to improve p99, revert during a quiet hour.
More recipes
- Fix Java Heap Space OOM in Kubernetes Containers
Size the JVM heap from the container limit with MaxRAMPercentage and leave room for native memory, so pods stop failing with heap errors or OOMKilled.
- Fix HikariCP Connection Is Not Available, Request Timed Out
Size the HikariCP connection pool in Spring Boot from your database limit and pod count, so requests stop timing out while they wait for a connection.
- Spring Boot Virtual Threads and server.tomcat.threads.max
Size server.tomcat.threads.max from your traffic, or turn on Spring Boot virtual threads, so requests stop queueing inside Tomcat at peak.
- Improve Cache Hit Ratio to Protect Your Origin at Peak
Trim the cache key, send cache headers from the origin and serve stale while revalidating, so the CDN absorbs peak traffic instead of passing it to your origin.