A RabbitMQ memory alarm stops every publisher in the cluster. One node crosses its memory watermark, and all nodes stop reading from publishing connections until it drops back. You set the watermark from the real memory limit below, keep queues from growing without bound and make your clients handle the block. Then you prove it under load on staging.
What breaks at peak
At peak, publishers outrun consumers and messages pile up. Once a node’s memory passes the watermark, it logs this:
memory resource limit alarm set on node rabbit@rabbitmq-0.
**********************************************************
*** Publishers will be blocked until this alarm clears ***
**********************************************************
It logs memory resource limit alarm cleared on node rabbit@rabbitmq-0 when memory falls again. In between, four things happen:
- The block is cluster-wide. One node over the watermark blocks publishers on every node.
- Publishers hang. The node stops reading from publishing connections. RabbitMQ stops checking their heartbeats, and
rabbitmqctl list_connections name stateshows them asblocked. Publish calls wait on TCP back pressure until they time out. Your API requests that publish an event hang with them. - Consumers on the same connection stall too. A connection that publishes and consumes gets blocked as a whole. Its consumers can’t send acks, so the queue never drains, and the alarm can’t clear. That deadlock turns a short spike into an outage.
- Consumers on their own connections keep working. Deliveries continue, and the alarm clears once they free enough memory.
The container trap. RabbitMQ computes its limit from the memory it detects. On Linux it reads /proc/meminfo, which reports the host’s RAM inside a container. A relative watermark of 0.6 on a 64 GiB node gives a 38 GiB limit, in a pod limited to 4 GiB. The alarm never fires. The kernel OOM-kills the node instead, with exit code 137, and its queues fail over or go offline. RabbitMQ’s own docs recommend an absolute limit in containers for this reason. The OOMKilled recipe covers setting the container limit itself from real usage. If you run Kafka, the Kafka consumer lag recipe covers the backlog side of a queue that can’t keep up.
The watermark marks where RabbitMQ throttles publishers. Memory can pass it while the alarm takes effect, so leave headroom below the container limit.
The vm_memory_high_watermark config
This example suits a node in a container with a 4 GiB memory limit. Changed lines only, in rabbitmq.conf:
vm_memory_high_watermark.absolute = 2400MiB
disk_free_limit.absolute = 4GB
Size the watermark from the limit. I start at 0.6 times the container limit, the same ratio RabbitMQ 4.0 applies to detected RAM, and round down. That leaves room for the overshoot, garbage collection and the kernel page cache. If you’d rather keep a relative value, set total_memory_available_override_value = 4GiB and vm_memory_high_watermark.relative = 0.6. The RabbitMQ Cluster Operator sets the limit from the pod’s resources for you, so check its config before you add your own.
disk_free_limit sets the free disk space below which a second alarm blocks publishers. Classic queues page messages to disk under pressure, so a tight disk turns a memory problem into a disk alarm. RabbitMQ’s production checklist recommends a disk limit about the size of the memory watermark. The 50MB default suits only development.
Apply the watermark to a running node to test it. The change lasts until the node restarts:
rabbitmqctl set_vm_memory_high_watermark absolute "2400MiB"
Find what holds the memory. Before you raise anything, look at the breakdown on the node that alarmed:
rabbitmq-diagnostics memory_breakdown
rabbitmq-diagnostics alarms
Large queue_procs or quorum_queue_procs means messages are piling up. Large connection_readers, connection_writers and connection_channels mean too many connections or channels, often from clients that open one per request. Large binary points at big messages or many unacknowledged ones held by a high prefetch.
Cap queue length. A queue with no limit grows until the alarm stops it, and then everyone waits. A length limit makes the queue refuse new messages instead:
rabbitmqctl set_policy --apply-to queues queue-limits "^orders\." \
'{"max-length":500000,"overflow":"reject-publish"}'
max-length caps the ready messages per queue. Size it from your memory budget: watermark divided by queue count divided by average message size, with room for unacked messages. Add max-length-bytes when message sizes vary. overflow: reject-publish refuses new messages once the queue is full. With publisher confirms on, the publisher gets a basic.nack for each one and can back off or fail the request. The other choice, the default drop-head, deletes the oldest messages without telling anyone. I use reject-publish for any queue whose messages matter.
Since 3.12, classic queues keep most messages on disk and ignore the old lazy mode. Quorum queues also write messages to disk and keep less per message in memory. The length limit still matters for both, because index entries, unacked messages and per-queue processes all take memory.
Fix the clients. Three changes keep a block from turning into a hang:
- Separate connections. Publish on one connection and consume on another, so a blocked publisher never stops acks.
- Bounded prefetch. Set
basic.qosto a small number per consumer, enough to keep it busy. An unlimited prefetch pushes every ready message into consumer buffers and RabbitMQ memory. - Confirms and a blocked timeout. Turn on publisher confirms and treat a nack as back pressure. Give the publishing connection a limit on how long it may stay blocked.
In Python with pika, that looks like this:
import os
import time
import pika
from pika.exceptions import ConnectionBlockedTimeout, NackError, UnroutableError
params = pika.URLParameters(os.environ["AMQP_URL"])
params.heartbeat = 30
params.blocked_connection_timeout = 20 # give up after 20 s blocked
publish_conn = pika.BlockingConnection(params)
consume_conn = pika.BlockingConnection(params) # never share with publishing
pub = publish_conn.channel()
pub.confirm_delivery()
sub = consume_conn.channel()
sub.basic_qos(prefetch_count=50)
def publish(body: bytes, attempts: int = 5) -> None:
for attempt in range(attempts):
try:
pub.basic_publish(
exchange="orders",
routing_key="order.created",
body=body,
properties=pika.BasicProperties(delivery_mode=2),
mandatory=True,
)
return
except NackError:
time.sleep(min(2 ** attempt, 10)) # queue full or broker under pressure
raise RuntimeError("broker refused the message after retries")
blocked_connection_timeout tears the connection down with ConnectionBlockedTimeout once it stays blocked that long. Catch it at the request level and return an error, rather than holding the request open. UnroutableError means no queue matched the routing key, which is a bug to log, not a reason to retry. In Java, register a BlockedListener on the connection. In Node.js, amqplib connections emit blocked and unblocked events.
Amazon MQ for RabbitMQ. Amazon MQ sets the memory limit from the broker instance type, and you can’t change it. CloudWatch shows RabbitMQMemUsed against RabbitMQMemLimit. While the alarm holds, Amazon MQ refuses a RebootBroker call, so a restart isn’t an escape hatch. The fixes there are the same: queue length policies, fewer connections and channels, a small prefetch and publisher confirms. After that, a larger instance type raises the limit.
Test it on your system
Use a staging cluster with production’s node count, container limits, queue types and policies. Drive it with RabbitMQ’s PerfTest. Pin publishers to a fixed rate above the consumers’ rate, so messages build up the way they do at peak:
java -jar perf-test.jar --uri amqp://user:pass@rabbitmq-staging:5672 \
--producers 4 --consumers 2 \
--rate 2000 --consumer-rate 1000 \
--size 4000 --flag persistent --confirm 100 --qos 50 \
--queue-pattern 'orders.perf-%d' --queue-pattern-from 1 --queue-pattern-to 4 \
--auto-delete false --time 1200
--rate is per producer and --consumer-rate per consumer, so this publishes 8,000 a second and consumes 2,000. Set them from your own peak publish rate and your real consumers’ throughput. The queue names match the ^orders\. policy, so the length limit applies. Add --quorum-queue if production uses quorum queues.
While it runs, watch memory against the watermark and the connection states:
watch -n 5 'rabbitmq-diagnostics -q status | grep -A3 -i "memory"; rabbitmq-diagnostics -q alarms'
rabbitmqctl list_connections name state
Watch the pod’s container_memory_working_set_bytes against its limit as well.
Decide what passes before you start. I use these:
- No node restarts and none is OOM-killed, and container memory stays under your chosen share of the limit.
- The queues stop at
max-length, and PerfTest reports nacked publishes instead of a memory alarm. - If the alarm does fire, consumers keep acking and the alarm clears without any restart.
- Your app’s publishing endpoint returns errors within its timeout and never hangs.
Run it a second time with the length policy removed to see the alarm. Confirm that your consumers drain the queues and the alarm clears. If it doesn’t clear, look for consumers that share a connection with a publisher.
Keep it healthy
Enable the rabbitmq_prometheus plugin, which serves metrics on port 15692, and scrape every node. The last rule also needs cAdvisor and kube-state-metrics with matching namespace and pod labels.
groups:
- name: rabbitmq-memory
rules:
- alert: RabbitMQMemoryAlarm
expr: max by (instance) (rabbitmq_alarms_memory_used_watermark) == 1
for: 1m
labels:
severity: critical
annotations:
summary: "Memory alarm on {{ $labels.instance }}, publishers are blocked cluster-wide"
description: "Run rabbitmq-diagnostics memory_breakdown on this node. Check consumers are running and acking, and that no consumer shares a connection with a publisher."
- alert: RabbitMQMemoryNearWatermark
expr: rabbitmq_process_resident_memory_bytes / rabbitmq_resident_memory_limit_bytes > 0.8
for: 5m
labels:
severity: warning
annotations:
summary: "RabbitMQ {{ $labels.instance }} above 80% of its memory watermark"
description: "Queues are growing or connections piling up. Check queue depth and consumer count, and confirm max-length policies cover the busy queues."
- alert: RabbitMQDiskAlarm
expr: max by (instance) (rabbitmq_alarms_free_disk_space_watermark) == 1
for: 1m
labels:
severity: critical
annotations:
summary: "Disk alarm on {{ $labels.instance }}, publishers are blocked cluster-wide"
description: "Free disk fell below disk_free_limit. Drain or purge the largest queues and expand the volume."
- alert: RabbitMQBacklogGrowing
expr: |
sum by (instance) (rabbitmq_queue_messages_ready) > 100000
and delta(sum by (instance) (rabbitmq_queue_messages_ready)[15m:1m]) > 0
for: 15m
labels:
severity: warning
annotations:
summary: "Ready messages on {{ $labels.instance }} keep growing"
description: "Publishers outrun consumers. Scale consumers, check their error rate and downstream latency, and check the queue length policy before memory reaches the watermark."
- alert: RabbitMQWatermarkAboveContainerLimit
expr: |
max by (namespace, pod) (rabbitmq_resident_memory_limit_bytes)
> on (namespace, pod)
0.8 * max by (namespace, pod) (kube_pod_container_resource_limits{resource="memory", container="rabbitmq"})
for: 30m
labels:
severity: warning
annotations:
summary: "RabbitMQ in {{ $labels.pod }} has a watermark too close to its container limit"
description: "The node would be OOM-killed before the alarm fires. Set vm_memory_high_watermark.absolute to about 0.6 of the container limit."
Set the 100000 backlog floor from your normal peak depth. The queue metrics here are per node, since rabbitmq_prometheus aggregates queue metrics unless you turn on per-object metrics. Also alert from your app on publish timeouts and ConnectionBlockedTimeout. Those reach you before the dashboard does.
Roll back
Save the current state first:
rabbitmq-diagnostics status > rabbitmq-status-before.txt
rabbitmqctl list_policies > rabbitmq-policies-before.txt
cp /etc/rabbitmq/rabbitmq.conf rabbitmq.conf.before
Restore the old rabbitmq.conf and restart nodes one at a time, waiting for each to rejoin before the next. For a quick revert of the watermark without a restart, run rabbitmqctl set_vm_memory_high_watermark with the old value on each node. Remove the length limit with rabbitmqctl clear_policy queue-limits.
If the length policy rejects messages your app can’t afford to lose, roll the policy back now and add consumers. If a lower watermark makes the alarm fire at normal load, raise it now, because every alarm blocks every publisher. For other reasons, do rolling restarts in a quiet window, since each one moves connections and queue leaders.
More recipes
- Improve Cache Hit Ratio to Protect Your Origin at Peak
Trim the cache key, send cache headers from the origin and serve stale while revalidating, so the CDN absorbs peak traffic instead of passing it to your origin.
- Fix ALB 502 Bad Gateway and 504 Gateway Timeout
Set each target's keep-alive above the ALB idle timeout and keep slow requests under it, so the load balancer stops answering peak traffic with 502 and 504.
- Fix Lambda TooManyRequestsException Rate Exceeded
Size Lambda concurrency from peak traffic and split it with reserved and provisioned concurrency, so critical functions stop throttling when traffic spikes.
- NAT Gateway ErrorPortAllocation and Port Exhaustion
Alert on ErrorPortAllocation and the signals that come before it, so you catch NAT gateway port exhaustion before outbound calls fail at peak.