Kafka consumer lag is the number of messages written to a partition that a consumer group hasn’t committed yet. Broker dashboards don’t show it. Bytes in, bytes out and under-replicated partitions all look healthy while a consumer group falls an hour behind. You export lag per group below, alert on what it means for your users and prove each alert fires.
What you miss today
Lag lives in consumer group offsets, which the brokers store but don’t chart. The client’s own records-lag-max metric sits in each consumer’s JMX, and it disappears the moment that consumer stops. A consumer that crashes, hangs or never starts reports no lag at all.
Your users find it first. Orders sit unconfirmed, emails go out late and search results miss the last hour. Then retention runs. Once the broker deletes segments past retention.ms that a group never read, the consumer logs this and jumps to auto.offset.reset:
Fetch position FetchPosition{offset=..., ...} is out of range for partition orders-3, resetting offset
With auto.offset.reset=latest, those messages are gone. With earliest, the group starts from the oldest data left.
Three patterns hide behind a healthy total:
- One hot partition. Your key gives one partition far more traffic than the rest. Its lag grows while total lag looks flat, because the other partitions sit near zero.
- Rebalance loops. A consumer that takes longer than
max.poll.interval.ms, 300000 by default, between polls gets kicked out. The client logsconsumer poll timeout has expired.Its next commit fails withCommitFailedException: Commit cannot be completed since the group has already rebalanced and assigned the partitions to another member.The group rebalances, the slow batch repeats, and lag grows in steps. - A ceiling on consumers. A group can’t use more consumers than the topic has partitions. Extra pods sit idle, so scaling out stops helping at that count.
Raw lag in messages also misleads. Ten thousand messages is nothing on a topic that takes 50,000 a second, and an hour of backlog on one that takes 3. You care how long the group needs to catch up.
The exporter and metrics
Two open source exporters read group offsets from the brokers. kafka_exporter is the long-standing one. KMinion adds end-to-end probes. Run one, not both. The rules below use kafka_exporter:
kafka_exporter \
--kafka.server=kafka-0.kafka:9092 \
--kafka.server=kafka-1.kafka:9092 \
--kafka.server=kafka-2.kafka:9092 \
--group.filter='^(orders|billing|search-indexer)-.*'
--group.filter takes a regex. Limit it to the groups you own, since every group and partition adds series. Add the --sasl.* or --tls.* flags your cluster needs.
| kafka_exporter metric | KMinion equivalent | What it tells you |
|---|---|---|
kafka_consumergroup_lag{consumergroup,topic,partition} | kminion_kafka_consumer_group_topic_partition_lag | Lag per partition. -1 means the group has no committed offset there. |
kafka_consumergroup_lag_sum{consumergroup,topic} | kminion_kafka_consumer_group_topic_lag | Total lag per group and topic. |
kafka_consumergroup_current_offset_sum{consumergroup,topic} | kminion_kafka_consumer_group_topic_offset_sum | Sum of committed offsets. Its rate is the group’s consume rate. |
kafka_topic_partition_current_offset{topic,partition} | kminion_kafka_topic_partition_high_water_mark | The log end offset. Its rate is the produce rate. |
kafka_topic_partition_oldest_offset{topic,partition} | kminion_kafka_topic_partition_low_water_mark | The oldest offset still on disk. Retention moves it. |
kafka_consumergroup_members{consumergroup} | kminion_kafka_consumer_group_members | Live consumers in the group. Zero means nobody is reading. |
KMinion labels its series group_id, topic_name and partition_id. Keep its minion.consumerGroups.granularity at partition, the default, to get per-partition lag.
The useful number is time to catch up: lag divided by the rate the group commits. The rules compute it from the table above, so you don’t need a time-lag exporter.
Managed Kafka. Amazon MSK publishes MaxOffsetLag, SumOffsetLag, OffsetLag, EstimatedTimeLag and EstimatedMaxTimeLag per group to CloudWatch, or through open monitoring with Prometheus. MSK emits them only for groups in a STABLE or EMPTY state with ASCII names and committed offsets, so a group stuck rebalancing goes quiet. Azure Event Hubs reports ConsumerLag in its application metrics logs on Premium and Dedicated only, as commit lag, evaluated once a minute. On Standard you compute lag yourself. Confluent Cloud’s Metrics API exposes io.confluent.kafka.server/consumer_lag_offsets, but consumers that use assign() report no lag. kafka_exporter and KMinion work against MSK and Confluent Cloud too, as long as they can authenticate.
The alert rules
groups:
- name: kafka-consumer-lag
rules:
- alert: KafkaConsumerCatchUpTooSlow
expr: |
sum by (consumergroup, topic) (kafka_consumergroup_lag_sum)
/
clamp_min(sum by (consumergroup, topic) (rate(kafka_consumergroup_current_offset_sum[5m])), 1)
> 600
for: 10m
labels:
severity: warning
annotations:
summary: "{{ $labels.consumergroup }} needs over 10 minutes to catch up on {{ $labels.topic }}"
description: "Compare the produce rate with the consume rate. Scale consumers up to the partition count, check the consumer's own latency to its database, or add partitions if every consumer is busy."
- alert: KafkaConsumerGroupStalled
expr: |
sum by (consumergroup, topic) (rate(kafka_consumergroup_current_offset_sum[10m])) == 0
and
sum by (consumergroup, topic) (kafka_consumergroup_lag_sum) > 0
for: 10m
labels:
severity: critical
annotations:
summary: "{{ $labels.consumergroup }} has stopped committing on {{ $labels.topic }}"
description: "Lag exists and no offset has moved in 10 minutes. Check that consumer pods run, read their logs for poll timeout and CommitFailedException, and look for a poison message at the stuck offset."
- alert: KafkaConsumerGroupEmpty
expr: kafka_consumergroup_members == 0
for: 5m
labels:
severity: critical
annotations:
summary: "Consumer group {{ $labels.consumergroup }} has no members"
description: "Nobody reads this group's topics. Check the consumer Deployment, its crash logs and its credentials. Remove the group from --group.filter if it is retired."
- alert: KafkaHotPartitionLag
expr: |
max by (consumergroup, topic) (kafka_consumergroup_lag)
> 0.5 * sum by (consumergroup, topic) (kafka_consumergroup_lag_sum)
and sum by (consumergroup, topic) (kafka_consumergroup_lag_sum) > 10000
for: 15m
labels:
severity: warning
annotations:
summary: "One partition holds most of the lag for {{ $labels.consumergroup }} on {{ $labels.topic }}"
description: "The partition key is skewed or one consumer is slow. Find the partition with kafka-consumer-groups.sh --describe, check which consumer owns it and review the message key."
- alert: KafkaConsumerBehindRetention
expr: |
(
kafka_consumergroup_current_offset
- on (topic, partition) group_left ()
kafka_topic_partition_oldest_offset
) < 0
and kafka_consumergroup_current_offset >= 0
for: 0m
labels:
severity: critical
annotations:
summary: "{{ $labels.consumergroup }} fell behind retention on {{ $labels.topic }} partition {{ $labels.partition }}"
description: "The broker deleted messages this group never read. The consumer resets per auto.offset.reset. Replay from a source of truth if you can, then raise retention.ms or consumer capacity."
The 600-second threshold in KafkaConsumerCatchUpTooSlow is a placeholder. Set it from how stale your users can tolerate the data, as an SLO, and see the tuning section below. clamp_min(..., 1) keeps the division defined when the group commits nothing, so a stalled group with lag fires this rule too.
KafkaConsumerBehindRetention uses for: 0m, because deleted messages don’t come back. The >= 0 filter skips partitions where the group never committed.
Test the alerts
Use a staging cluster with production’s partition counts and a consumer deployment that runs your real code. Produce at a fixed rate with Kafka’s perf tool, which holds --throughput messages per second whatever the consumers do:
kafka-producer-perf-test.sh --topic orders \
--num-records 6000000 --record-size 1024 --throughput 5000 \
--producer-props bootstrap.servers=kafka-0.kafka:9092 acks=all
Set --throughput to your peak produce rate and --record-size to your average message. While it runs, read the group the same way the exporter does:
kafka-consumer-groups.sh --bootstrap-server kafka-0.kafka:9092 \
--describe --group orders-consumer
Then break the consumer in four ways, one at a time:
- Stop it.
kubectl scale deployment orders-consumer --replicas=0.KafkaConsumerGroupEmptyfires after 5 minutes, thenKafkaConsumerGroupStalledandKafkaConsumerCatchUpTooSlow. - Slow it. Scale to one replica, or add a sleep per message behind a feature flag. Lag grows without pause and
KafkaConsumerCatchUpTooSlowfires whileStalledstays quiet. - Skew it. Produce with one fixed key, using your own producer or a script, so every message lands on one partition.
KafkaHotPartitionLagfires. - Outrun retention. On a test topic, set
retention.ms=60000, stop the consumer and keep producing. Within a few minutesKafkaConsumerBehindRetentionfires, and the consumer logsis out of range for partitionwhen it restarts.
Restore the consumer after each test and confirm the alert clears.
The recovery also sizes your partitions. Watch rate(kafka_consumergroup_current_offset_sum[1m]) while the group drains a backlog, since a busy consumer runs flat out then. Divide that rate by the consumer count to get one consumer’s throughput on your code. Your partition count must be at least your peak produce rate divided by that number, with headroom for the peak after a deploy. Adding partitions later changes which partition each key maps to, so decide before you rely on per-key ordering.
Tune the noise
Set the catch-up threshold from how stale your data may get. If your users accept search results 5 minutes old, alert when a group needs more than about half of that to catch up. You then have time to scale before the SLO breaks.
Lag climbs at every deploy, rebalance and batch job, then drains. Look at a week of sum by (consumergroup, topic) (kafka_consumergroup_lag_sum) and note how long normal spikes last. Set for longer than those. A 10-minute for on catch-up time skips a rolling restart and still pages on a real backlog.
Give each group its own threshold when their needs differ. A billing group that must stay within a minute and an analytics loader that runs hourly can’t share one number. Put the threshold in a label on a recording rule, or split the rule by consumergroup. I’d rather run five rules with honest thresholds than one rule everyone mutes.
Skip KafkaHotPartitionLag on topics with a handful of partitions, where one partition holding half the lag is normal. Raise its floor of 10000 messages on high-volume topics.
To scale consumers on lag instead of CPU, feed lag or catch-up time to KEDA. The HPA recipe shows that setup for queue depth. Cap replicas at the partition count. RabbitMQ fails differently at peak: it blocks publishers when memory runs short, which the RabbitMQ memory alarm recipe covers.
Batch consumers that run on a schedule look stalled between runs. Exclude them from KafkaConsumerGroupStalled with a consumergroup!~ matcher, and alert on their run finishing instead.
More recipes
- Improve Cache Hit Ratio to Protect Your Origin at Peak
Trim the cache key, send cache headers from the origin and serve stale while revalidating, so the CDN absorbs peak traffic instead of passing it to your origin.
- Fix ALB 502 Bad Gateway and 504 Gateway Timeout
Set each target's keep-alive above the ALB idle timeout and keep slow requests under it, so the load balancer stops answering peak traffic with 502 and 504.
- Fix Lambda TooManyRequestsException Rate Exceeded
Size Lambda concurrency from peak traffic and split it with reserved and provisioned concurrency, so critical functions stop throttling when traffic spikes.
- NAT Gateway ErrorPortAllocation and Port Exhaustion
Alert on ErrorPortAllocation and the signals that come before it, so you catch NAT gateway port exhaustion before outbound calls fail at peak.