Kafka Consumer Lag Monitoring with Prometheus Alerts

Alert on how long each consumer group needs to catch up, on stalled groups and on retention, so Kafka consumer lag pages you before data goes stale.

Published

Applies to Apache Kafka 3.x and 4.x with offsets stored in Kafka; current kafka_exporter and KMinion releases

Kafka consumer lag is the number of messages written to a partition that a consumer group hasn’t committed yet. Broker dashboards don’t show it. Bytes in, bytes out and under-replicated partitions all look healthy while a consumer group falls an hour behind. You export lag per group below, alert on what it means for your users and prove each alert fires.

What you miss today

Lag lives in consumer group offsets, which the brokers store but don’t chart. The client’s own records-lag-max metric sits in each consumer’s JMX, and it disappears the moment that consumer stops. A consumer that crashes, hangs or never starts reports no lag at all.

Your users find it first. Orders sit unconfirmed, emails go out late and search results miss the last hour. Then retention runs. Once the broker deletes segments past retention.ms that a group never read, the consumer logs this and jumps to auto.offset.reset:

Fetch position FetchPosition{offset=..., ...} is out of range for partition orders-3, resetting offset

With auto.offset.reset=latest, those messages are gone. With earliest, the group starts from the oldest data left.

Three patterns hide behind a healthy total:

  • One hot partition. Your key gives one partition far more traffic than the rest. Its lag grows while total lag looks flat, because the other partitions sit near zero.
  • Rebalance loops. A consumer that takes longer than max.poll.interval.ms, 300000 by default, between polls gets kicked out. The client logs consumer poll timeout has expired. Its next commit fails with CommitFailedException: Commit cannot be completed since the group has already rebalanced and assigned the partitions to another member. The group rebalances, the slow batch repeats, and lag grows in steps.
  • A ceiling on consumers. A group can’t use more consumers than the topic has partitions. Extra pods sit idle, so scaling out stops helping at that count.

Raw lag in messages also misleads. Ten thousand messages is nothing on a topic that takes 50,000 a second, and an hour of backlog on one that takes 3. You care how long the group needs to catch up.

The exporter and metrics

Two open source exporters read group offsets from the brokers. kafka_exporter is the long-standing one. KMinion adds end-to-end probes. Run one, not both. The rules below use kafka_exporter:

kafka_exporter \
  --kafka.server=kafka-0.kafka:9092 \
  --kafka.server=kafka-1.kafka:9092 \
  --kafka.server=kafka-2.kafka:9092 \
  --group.filter='^(orders|billing|search-indexer)-.*'

--group.filter takes a regex. Limit it to the groups you own, since every group and partition adds series. Add the --sasl.* or --tls.* flags your cluster needs.

kafka_exporter metricKMinion equivalentWhat it tells you
kafka_consumergroup_lag{consumergroup,topic,partition}kminion_kafka_consumer_group_topic_partition_lagLag per partition. -1 means the group has no committed offset there.
kafka_consumergroup_lag_sum{consumergroup,topic}kminion_kafka_consumer_group_topic_lagTotal lag per group and topic.
kafka_consumergroup_current_offset_sum{consumergroup,topic}kminion_kafka_consumer_group_topic_offset_sumSum of committed offsets. Its rate is the group’s consume rate.
kafka_topic_partition_current_offset{topic,partition}kminion_kafka_topic_partition_high_water_markThe log end offset. Its rate is the produce rate.
kafka_topic_partition_oldest_offset{topic,partition}kminion_kafka_topic_partition_low_water_markThe oldest offset still on disk. Retention moves it.
kafka_consumergroup_members{consumergroup}kminion_kafka_consumer_group_membersLive consumers in the group. Zero means nobody is reading.

KMinion labels its series group_id, topic_name and partition_id. Keep its minion.consumerGroups.granularity at partition, the default, to get per-partition lag.

The useful number is time to catch up: lag divided by the rate the group commits. The rules compute it from the table above, so you don’t need a time-lag exporter.

Managed Kafka. Amazon MSK publishes MaxOffsetLag, SumOffsetLag, OffsetLag, EstimatedTimeLag and EstimatedMaxTimeLag per group to CloudWatch, or through open monitoring with Prometheus. MSK emits them only for groups in a STABLE or EMPTY state with ASCII names and committed offsets, so a group stuck rebalancing goes quiet. Azure Event Hubs reports ConsumerLag in its application metrics logs on Premium and Dedicated only, as commit lag, evaluated once a minute. On Standard you compute lag yourself. Confluent Cloud’s Metrics API exposes io.confluent.kafka.server/consumer_lag_offsets, but consumers that use assign() report no lag. kafka_exporter and KMinion work against MSK and Confluent Cloud too, as long as they can authenticate.

The alert rules

groups:
  - name: kafka-consumer-lag
    rules:
      - alert: KafkaConsumerCatchUpTooSlow
        expr: |
          sum by (consumergroup, topic) (kafka_consumergroup_lag_sum)
            /
          clamp_min(sum by (consumergroup, topic) (rate(kafka_consumergroup_current_offset_sum[5m])), 1)
            > 600
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "{{ $labels.consumergroup }} needs over 10 minutes to catch up on {{ $labels.topic }}"
          description: "Compare the produce rate with the consume rate. Scale consumers up to the partition count, check the consumer's own latency to its database, or add partitions if every consumer is busy."
      - alert: KafkaConsumerGroupStalled
        expr: |
          sum by (consumergroup, topic) (rate(kafka_consumergroup_current_offset_sum[10m])) == 0
            and
          sum by (consumergroup, topic) (kafka_consumergroup_lag_sum) > 0
        for: 10m
        labels:
          severity: critical
        annotations:
          summary: "{{ $labels.consumergroup }} has stopped committing on {{ $labels.topic }}"
          description: "Lag exists and no offset has moved in 10 minutes. Check that consumer pods run, read their logs for poll timeout and CommitFailedException, and look for a poison message at the stuck offset."
      - alert: KafkaConsumerGroupEmpty
        expr: kafka_consumergroup_members == 0
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Consumer group {{ $labels.consumergroup }} has no members"
          description: "Nobody reads this group's topics. Check the consumer Deployment, its crash logs and its credentials. Remove the group from --group.filter if it is retired."
      - alert: KafkaHotPartitionLag
        expr: |
          max by (consumergroup, topic) (kafka_consumergroup_lag)
            > 0.5 * sum by (consumergroup, topic) (kafka_consumergroup_lag_sum)
            and sum by (consumergroup, topic) (kafka_consumergroup_lag_sum) > 10000
        for: 15m
        labels:
          severity: warning
        annotations:
          summary: "One partition holds most of the lag for {{ $labels.consumergroup }} on {{ $labels.topic }}"
          description: "The partition key is skewed or one consumer is slow. Find the partition with kafka-consumer-groups.sh --describe, check which consumer owns it and review the message key."
      - alert: KafkaConsumerBehindRetention
        expr: |
          (
            kafka_consumergroup_current_offset
              - on (topic, partition) group_left ()
            kafka_topic_partition_oldest_offset
          ) < 0
            and kafka_consumergroup_current_offset >= 0
        for: 0m
        labels:
          severity: critical
        annotations:
          summary: "{{ $labels.consumergroup }} fell behind retention on {{ $labels.topic }} partition {{ $labels.partition }}"
          description: "The broker deleted messages this group never read. The consumer resets per auto.offset.reset. Replay from a source of truth if you can, then raise retention.ms or consumer capacity."

The 600-second threshold in KafkaConsumerCatchUpTooSlow is a placeholder. Set it from how stale your users can tolerate the data, as an SLO, and see the tuning section below. clamp_min(..., 1) keeps the division defined when the group commits nothing, so a stalled group with lag fires this rule too.

KafkaConsumerBehindRetention uses for: 0m, because deleted messages don’t come back. The >= 0 filter skips partitions where the group never committed.

Test the alerts

Use a staging cluster with production’s partition counts and a consumer deployment that runs your real code. Produce at a fixed rate with Kafka’s perf tool, which holds --throughput messages per second whatever the consumers do:

kafka-producer-perf-test.sh --topic orders \
  --num-records 6000000 --record-size 1024 --throughput 5000 \
  --producer-props bootstrap.servers=kafka-0.kafka:9092 acks=all

Set --throughput to your peak produce rate and --record-size to your average message. While it runs, read the group the same way the exporter does:

kafka-consumer-groups.sh --bootstrap-server kafka-0.kafka:9092 \
  --describe --group orders-consumer

Then break the consumer in four ways, one at a time:

  1. Stop it. kubectl scale deployment orders-consumer --replicas=0. KafkaConsumerGroupEmpty fires after 5 minutes, then KafkaConsumerGroupStalled and KafkaConsumerCatchUpTooSlow.
  2. Slow it. Scale to one replica, or add a sleep per message behind a feature flag. Lag grows without pause and KafkaConsumerCatchUpTooSlow fires while Stalled stays quiet.
  3. Skew it. Produce with one fixed key, using your own producer or a script, so every message lands on one partition. KafkaHotPartitionLag fires.
  4. Outrun retention. On a test topic, set retention.ms=60000, stop the consumer and keep producing. Within a few minutes KafkaConsumerBehindRetention fires, and the consumer logs is out of range for partition when it restarts.

Restore the consumer after each test and confirm the alert clears.

The recovery also sizes your partitions. Watch rate(kafka_consumergroup_current_offset_sum[1m]) while the group drains a backlog, since a busy consumer runs flat out then. Divide that rate by the consumer count to get one consumer’s throughput on your code. Your partition count must be at least your peak produce rate divided by that number, with headroom for the peak after a deploy. Adding partitions later changes which partition each key maps to, so decide before you rely on per-key ordering.

Tune the noise

Set the catch-up threshold from how stale your data may get. If your users accept search results 5 minutes old, alert when a group needs more than about half of that to catch up. You then have time to scale before the SLO breaks.

Lag climbs at every deploy, rebalance and batch job, then drains. Look at a week of sum by (consumergroup, topic) (kafka_consumergroup_lag_sum) and note how long normal spikes last. Set for longer than those. A 10-minute for on catch-up time skips a rolling restart and still pages on a real backlog.

Give each group its own threshold when their needs differ. A billing group that must stay within a minute and an analytics loader that runs hourly can’t share one number. Put the threshold in a label on a recording rule, or split the rule by consumergroup. I’d rather run five rules with honest thresholds than one rule everyone mutes.

Skip KafkaHotPartitionLag on topics with a handful of partitions, where one partition holding half the lag is normal. Raise its floor of 10000 messages on high-volume topics.

To scale consumers on lag instead of CPU, feed lag or catch-up time to KEDA. The HPA recipe shows that setup for queue depth. Cap replicas at the partition count. RabbitMQ fails differently at peak: it blocks publishers when memory runs short, which the RabbitMQ memory alarm recipe covers.

Batch consumers that run on a schedule look stalled between runs. Exclude them from KafkaConsumerGroupStalled with a consumergroup!~ matcher, and alert on their run finishing instead.

More recipes

Further reading