Fix the RabbitMQ Memory Alarm and Blocked Publishers

Set vm_memory_high_watermark from the real memory limit and cap queue length, so a RabbitMQ memory alarm stays rare and publishers back off instead of hanging.

Published

Applies to RabbitMQ 3.13 and 4.x; the default watermark changed from 0.4 to 0.6 in 4.0

A RabbitMQ memory alarm stops every publisher in the cluster. One node crosses its memory watermark, and all nodes stop reading from publishing connections until it drops back. You set the watermark from the real memory limit below, keep queues from growing without bound and make your clients handle the block. Then you prove it under load on staging.

What breaks at peak

At peak, publishers outrun consumers and messages pile up. Once a node’s memory passes the watermark, it logs this:

memory resource limit alarm set on node rabbit@rabbitmq-0.

**********************************************************
*** Publishers will be blocked until this alarm clears ***
**********************************************************

It logs memory resource limit alarm cleared on node rabbit@rabbitmq-0 when memory falls again. In between, four things happen:

  • The block is cluster-wide. One node over the watermark blocks publishers on every node.
  • Publishers hang. The node stops reading from publishing connections. RabbitMQ stops checking their heartbeats, and rabbitmqctl list_connections name state shows them as blocked. Publish calls wait on TCP back pressure until they time out. Your API requests that publish an event hang with them.
  • Consumers on the same connection stall too. A connection that publishes and consumes gets blocked as a whole. Its consumers can’t send acks, so the queue never drains, and the alarm can’t clear. That deadlock turns a short spike into an outage.
  • Consumers on their own connections keep working. Deliveries continue, and the alarm clears once they free enough memory.

The container trap. RabbitMQ computes its limit from the memory it detects. On Linux it reads /proc/meminfo, which reports the host’s RAM inside a container. A relative watermark of 0.6 on a 64 GiB node gives a 38 GiB limit, in a pod limited to 4 GiB. The alarm never fires. The kernel OOM-kills the node instead, with exit code 137, and its queues fail over or go offline. RabbitMQ’s own docs recommend an absolute limit in containers for this reason. The OOMKilled recipe covers setting the container limit itself from real usage. If you run Kafka, the Kafka consumer lag recipe covers the backlog side of a queue that can’t keep up.

The watermark marks where RabbitMQ throttles publishers. Memory can pass it while the alarm takes effect, so leave headroom below the container limit.

The vm_memory_high_watermark config

This example suits a node in a container with a 4 GiB memory limit. Changed lines only, in rabbitmq.conf:

vm_memory_high_watermark.absolute = 2400MiB
disk_free_limit.absolute = 4GB

Size the watermark from the limit. I start at 0.6 times the container limit, the same ratio RabbitMQ 4.0 applies to detected RAM, and round down. That leaves room for the overshoot, garbage collection and the kernel page cache. If you’d rather keep a relative value, set total_memory_available_override_value = 4GiB and vm_memory_high_watermark.relative = 0.6. The RabbitMQ Cluster Operator sets the limit from the pod’s resources for you, so check its config before you add your own.

disk_free_limit sets the free disk space below which a second alarm blocks publishers. Classic queues page messages to disk under pressure, so a tight disk turns a memory problem into a disk alarm. RabbitMQ’s production checklist recommends a disk limit about the size of the memory watermark. The 50MB default suits only development.

Apply the watermark to a running node to test it. The change lasts until the node restarts:

rabbitmqctl set_vm_memory_high_watermark absolute "2400MiB"

Find what holds the memory. Before you raise anything, look at the breakdown on the node that alarmed:

rabbitmq-diagnostics memory_breakdown
rabbitmq-diagnostics alarms

Large queue_procs or quorum_queue_procs means messages are piling up. Large connection_readers, connection_writers and connection_channels mean too many connections or channels, often from clients that open one per request. Large binary points at big messages or many unacknowledged ones held by a high prefetch.

Cap queue length. A queue with no limit grows until the alarm stops it, and then everyone waits. A length limit makes the queue refuse new messages instead:

rabbitmqctl set_policy --apply-to queues queue-limits "^orders\." \
  '{"max-length":500000,"overflow":"reject-publish"}'

max-length caps the ready messages per queue. Size it from your memory budget: watermark divided by queue count divided by average message size, with room for unacked messages. Add max-length-bytes when message sizes vary. overflow: reject-publish refuses new messages once the queue is full. With publisher confirms on, the publisher gets a basic.nack for each one and can back off or fail the request. The other choice, the default drop-head, deletes the oldest messages without telling anyone. I use reject-publish for any queue whose messages matter.

Since 3.12, classic queues keep most messages on disk and ignore the old lazy mode. Quorum queues also write messages to disk and keep less per message in memory. The length limit still matters for both, because index entries, unacked messages and per-queue processes all take memory.

Fix the clients. Three changes keep a block from turning into a hang:

  • Separate connections. Publish on one connection and consume on another, so a blocked publisher never stops acks.
  • Bounded prefetch. Set basic.qos to a small number per consumer, enough to keep it busy. An unlimited prefetch pushes every ready message into consumer buffers and RabbitMQ memory.
  • Confirms and a blocked timeout. Turn on publisher confirms and treat a nack as back pressure. Give the publishing connection a limit on how long it may stay blocked.

In Python with pika, that looks like this:

import os
import time

import pika
from pika.exceptions import ConnectionBlockedTimeout, NackError, UnroutableError

params = pika.URLParameters(os.environ["AMQP_URL"])
params.heartbeat = 30
params.blocked_connection_timeout = 20  # give up after 20 s blocked

publish_conn = pika.BlockingConnection(params)
consume_conn = pika.BlockingConnection(params)  # never share with publishing

pub = publish_conn.channel()
pub.confirm_delivery()

sub = consume_conn.channel()
sub.basic_qos(prefetch_count=50)


def publish(body: bytes, attempts: int = 5) -> None:
    for attempt in range(attempts):
        try:
            pub.basic_publish(
                exchange="orders",
                routing_key="order.created",
                body=body,
                properties=pika.BasicProperties(delivery_mode=2),
                mandatory=True,
            )
            return
        except NackError:
            time.sleep(min(2 ** attempt, 10))  # queue full or broker under pressure
    raise RuntimeError("broker refused the message after retries")

blocked_connection_timeout tears the connection down with ConnectionBlockedTimeout once it stays blocked that long. Catch it at the request level and return an error, rather than holding the request open. UnroutableError means no queue matched the routing key, which is a bug to log, not a reason to retry. In Java, register a BlockedListener on the connection. In Node.js, amqplib connections emit blocked and unblocked events.

Amazon MQ for RabbitMQ. Amazon MQ sets the memory limit from the broker instance type, and you can’t change it. CloudWatch shows RabbitMQMemUsed against RabbitMQMemLimit. While the alarm holds, Amazon MQ refuses a RebootBroker call, so a restart isn’t an escape hatch. The fixes there are the same: queue length policies, fewer connections and channels, a small prefetch and publisher confirms. After that, a larger instance type raises the limit.

Test it on your system

Use a staging cluster with production’s node count, container limits, queue types and policies. Drive it with RabbitMQ’s PerfTest. Pin publishers to a fixed rate above the consumers’ rate, so messages build up the way they do at peak:

java -jar perf-test.jar --uri amqp://user:pass@rabbitmq-staging:5672 \
  --producers 4 --consumers 2 \
  --rate 2000 --consumer-rate 1000 \
  --size 4000 --flag persistent --confirm 100 --qos 50 \
  --queue-pattern 'orders.perf-%d' --queue-pattern-from 1 --queue-pattern-to 4 \
  --auto-delete false --time 1200

--rate is per producer and --consumer-rate per consumer, so this publishes 8,000 a second and consumes 2,000. Set them from your own peak publish rate and your real consumers’ throughput. The queue names match the ^orders\. policy, so the length limit applies. Add --quorum-queue if production uses quorum queues.

While it runs, watch memory against the watermark and the connection states:

watch -n 5 'rabbitmq-diagnostics -q status | grep -A3 -i "memory"; rabbitmq-diagnostics -q alarms'
rabbitmqctl list_connections name state

Watch the pod’s container_memory_working_set_bytes against its limit as well.

Decide what passes before you start. I use these:

  • No node restarts and none is OOM-killed, and container memory stays under your chosen share of the limit.
  • The queues stop at max-length, and PerfTest reports nacked publishes instead of a memory alarm.
  • If the alarm does fire, consumers keep acking and the alarm clears without any restart.
  • Your app’s publishing endpoint returns errors within its timeout and never hangs.

Run it a second time with the length policy removed to see the alarm. Confirm that your consumers drain the queues and the alarm clears. If it doesn’t clear, look for consumers that share a connection with a publisher.

Keep it healthy

Enable the rabbitmq_prometheus plugin, which serves metrics on port 15692, and scrape every node. The last rule also needs cAdvisor and kube-state-metrics with matching namespace and pod labels.

groups:
  - name: rabbitmq-memory
    rules:
      - alert: RabbitMQMemoryAlarm
        expr: max by (instance) (rabbitmq_alarms_memory_used_watermark) == 1
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "Memory alarm on {{ $labels.instance }}, publishers are blocked cluster-wide"
          description: "Run rabbitmq-diagnostics memory_breakdown on this node. Check consumers are running and acking, and that no consumer shares a connection with a publisher."
      - alert: RabbitMQMemoryNearWatermark
        expr: rabbitmq_process_resident_memory_bytes / rabbitmq_resident_memory_limit_bytes > 0.8
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "RabbitMQ {{ $labels.instance }} above 80% of its memory watermark"
          description: "Queues are growing or connections piling up. Check queue depth and consumer count, and confirm max-length policies cover the busy queues."
      - alert: RabbitMQDiskAlarm
        expr: max by (instance) (rabbitmq_alarms_free_disk_space_watermark) == 1
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "Disk alarm on {{ $labels.instance }}, publishers are blocked cluster-wide"
          description: "Free disk fell below disk_free_limit. Drain or purge the largest queues and expand the volume."
      - alert: RabbitMQBacklogGrowing
        expr: |
          sum by (instance) (rabbitmq_queue_messages_ready) > 100000
            and delta(sum by (instance) (rabbitmq_queue_messages_ready)[15m:1m]) > 0
        for: 15m
        labels:
          severity: warning
        annotations:
          summary: "Ready messages on {{ $labels.instance }} keep growing"
          description: "Publishers outrun consumers. Scale consumers, check their error rate and downstream latency, and check the queue length policy before memory reaches the watermark."
      - alert: RabbitMQWatermarkAboveContainerLimit
        expr: |
          max by (namespace, pod) (rabbitmq_resident_memory_limit_bytes)
            > on (namespace, pod)
          0.8 * max by (namespace, pod) (kube_pod_container_resource_limits{resource="memory", container="rabbitmq"})
        for: 30m
        labels:
          severity: warning
        annotations:
          summary: "RabbitMQ in {{ $labels.pod }} has a watermark too close to its container limit"
          description: "The node would be OOM-killed before the alarm fires. Set vm_memory_high_watermark.absolute to about 0.6 of the container limit."

Set the 100000 backlog floor from your normal peak depth. The queue metrics here are per node, since rabbitmq_prometheus aggregates queue metrics unless you turn on per-object metrics. Also alert from your app on publish timeouts and ConnectionBlockedTimeout. Those reach you before the dashboard does.

Roll back

Save the current state first:

rabbitmq-diagnostics status > rabbitmq-status-before.txt
rabbitmqctl list_policies > rabbitmq-policies-before.txt
cp /etc/rabbitmq/rabbitmq.conf rabbitmq.conf.before

Restore the old rabbitmq.conf and restart nodes one at a time, waiting for each to rejoin before the next. For a quick revert of the watermark without a restart, run rabbitmqctl set_vm_memory_high_watermark with the old value on each node. Remove the length limit with rabbitmqctl clear_policy queue-limits.

If the length policy rejects messages your app can’t afford to lose, roll the policy back now and add consumers. If a lower watermark makes the alarm fire at normal load, raise it now, because every alarm blocks every publisher. For other reasons, do rolling restarts in a quiet window, since each one moves connections and queue leaders.

More recipes

Further reading