RabbitMQ Monitoring: 6 Signals That Predict an Outage

A queue grew for three days before anyone noticed. Its single consumer had crashed on a Tuesday night. There was no exception in any app log someone was watching and no alert, just a channel that closed and never reopened. Publishers kept publishing. By Friday the queue held 2.1 million messages, the node hit its memory watermark, and every producer in the system blocked at once. The postmortem also turned up a dead letter queue with 14 months of messages in it. Nobody knew it existed.

That incident is what RabbitMQ monitoring is for. The goal isn't a prettier dashboard. It's catching the six failure patterns that broker outages are made of, days before they become outages. RabbitMQ is unusually honest about its internal state: nearly everything you need is exposed through rabbitmqctl, the management UI and the management HTTP API. The hard part is knowing which of the hundreds of numbers matter and what "bad" looks like for each.

This guide covers those six signals, how each failure develops, one detection step for each, and the impact when it's missed.

Part two is a hands-on DIY guide to RabbitMQ monitoring with native tools only. Part three covers putting an AI agent on top of the management API to watch and investigate continuously.

01Flow

Depth vs drain

Publish rate above deliver rate for long enough, and depth grows without bound.

02Flow

Unacked buildup

Consumers take deliveries and stop acking. Everything looks alive while throughput stops.

03Flow

Undrained DLQs

Failed business events land in a queue nobody consumes or alerts on.

04Broker

Resource alarms

Memory or disk crosses a limit and every publishing connection blocks.

05Broker

Quorum health

A cluster running one node short, one failure away from a write outage.

06Broker

Connection churn

A client opening a connection per message taxes CPU and file descriptors for everyone.

The six signals this guide covers. The first three are about messages, the last three about the broker itself.

Why the usual alerts miss these

Most RabbitMQ setups get monitored the way a web server does: is the process up, is the port open, is CPU sane. All three were fine for the whole three days in the opening story. Broker failures don't look like a host going down. They look like a ratio slowly going wrong.

Three structural reasons the standard setup misses them:

Static depth thresholds can't tell a burst from a leak. "Queue over 10,000" fires during every marketing email and stays silent while a dead consumer lets a low-traffic queue grow by 50 messages an hour.
The bad states look alive. A stuck consumer is still connected, still registered, still holding its channel. Every liveness check passes.
The damage lands somewhere else. When a resource alarm blocks publishers, the broker logs no error. Application timeouts fire in whichever service published last, and that's where the on-call starts digging.

Each signal below comes with the check that sees past those three problems.

1. Queue depth growing faster than consumers drain it

What it is. The basic equation of any queue: publish rate versus delivery rate. When publishing outpaces consuming for long enough, depth grows until memory or disk pressure stops the whole broker.

Burst or leak? You can't tell from a single depth number.

01Burst

Publish spikes

Deliver rate stays healthy. Depth climbs, then drains once the spike passes. Capacity planning.

02Leak

Drain drops

Publish rate is flat, deliver rate falls. Depth climbs in a straight line and never comes back. Incident.

Same depth graph, opposite causes. You need depth and both rates over time to tell them apart.

How to detect it. The quickest snapshot:

rabbitmqctl list_queues name messages messages_ready messages_unacknowledged consumers

Any queue with a growing messages count, a backlog, and consumers at 0 is the three-day incident waiting to happen. For rates, the management API exposes them per queue (%2f is the URL-encoded default vhost /):

curl -s -u monitoring:password \
  'http://localhost:15672/api/queues/%2f/orders' | \
  jq '{messages,
       publish: .message_stats.publish_details.rate,
       deliver: .message_stats.deliver_get_details.rate}'

If publish sits above deliver sample after sample, it's a leak. In the management UI, Queues → (queue) → Overview shows both rates on one chart.

Typical impact. A slow-motion outage. Hours of stale downstream data before anything visibly breaks, then a broker-wide failure when the backlog hits a resource limit. Whatever consumes that queue (order processing, notifications, billing events) falls behind first, silently.

2. Unacked messages piling up

What it is. messages_unacknowledged counts messages delivered to a consumer that hasn't acked them yet. A steady low number is normal pipelining. A high, flat number that doesn't move means consumers are taking deliveries and not finishing them. They're stuck on a dead downstream dependency, deadlocked, or holding a prefetch buffer far bigger than they can work through.

Why it happens. Two classic causes. First, prefetch set too high or unlimited: a consumer grabs 5,000 messages into memory, dies mid-batch, and all 5,000 get redelivered, possibly to the same broken consumer, in a loop. Second, a consumer that processes message 1 and then blocks forever on message 2, pinning everything already delivered to its channel.

How to detect it. Look at unacked counts per channel, next to prefetch:

rabbitmqctl list_channels connection name prefetch_count messages_unacknowledged

A channel where messages_unacknowledged equals prefetch_count and stays there is a stuck consumer. It has taken its full window and acked nothing since. In the management UI, the Channels tab shows the same columns, sortable. RabbitMQ's consumer prefetch docs cover sane values; for most workloads a prefetch between 30 and 300 beats both extremes.

Typical impact. Throughput collapses while every surface metric looks alive: connections up, consumers registered, queue "being consumed." Redelivery loops can also multiply side effects (double emails, double webhooks) if consumers aren't idempotent.

3. Dead letter queues nobody drains

What it is. A dead letter exchange catches messages that were rejected, expired, or overflowed a length limit. Wiring one up is good practice. But a DLQ is a landing zone, not a destination, and in most shops nothing consumes it and nobody looks at it. Every message in there is a real business event that failed: an order that didn't process, a payment webhook that got rejected, a provisioning job that never ran.

Why it happens. The team that added the DLX moved on. There's no consumer, no alert on depth, and no runbook for triage. The queue becomes a graveyard. A growing one is worse: it means something upstream is producing messages that consumers reject over and over, right now.

How to detect it. First find your dead-lettering topology. A DLX can be set by queue argument or by policy, so check both:

rabbitmqctl list_queues name messages policy arguments | grep -i dead-letter
rabbitmqctl list_policies

Then check depth on the DLQs themselves (they're ordinary queues, so the list_queues output includes them). Two numbers to alert on: depth above zero for more than a day, and rate of growth. A DLQ gaining messages every hour means an active failure, not old debris. In the management UI, dead-lettered rates show per queue under Queues → (queue) → Message rates.

Typical impact. Silent data loss with a paper trail nobody reads. Teams that open a year-old DLQ for the first time typically find hundreds to tens of thousands of lost business events, and, in the rejection reasons, a bug that's been live the whole time.

4. Memory and disk alarms that block publishers

What it is. When a node crosses its memory high watermark or free disk drops below disk_free_limit, RabbitMQ raises a resource alarm and blocks all connections that publish. Consumers keep running (the broker is trying to drain itself), but every producer stalls. From the application side it looks like publishes hanging forever, not an error.

Why it happens. Usually as the end stage of signal 1 or 3: a backlog someone ignored finally ate the node's memory. Disk alarms also fire on undersized VMs where logs or other services share the volume.

01Tuesday

Consumer dies

A channel closes and never reopens. No exception anyone is watching.

02Wednesday

Depth grows

Publishers keep publishing. Downstream data goes stale quietly.

03Friday

Memory alarm

The node crosses its watermark and the broker blocks publishers.

04Friday

Everything stops

Every producer hangs at once. Timeouts fire far from the cause.

Signals 1 and 4 are the same incident, three days apart. The cheap moment to catch it is the first cell.

How to detect it. One command tells you if you're blocked right now:

rabbitmq-diagnostics alarms

Check whether connections are in the blocked or blocking state with rabbitmqctl list_connections name state, and see what's eating memory with rabbitmq-diagnostics memory_breakdown. In the management UI, an active alarm turns the node red on the Overview page. That's visible only if someone has the page open, so alert on the alarm itself.

Typical impact. The "everything stopped at once" incident. Because publishers block silently instead of erroring, application timeouts fire far from the root cause, and the on-call spends the first 30 minutes debugging the wrong service.

5. Cluster and quorum queue health

What it is. In a cluster, quorum queues replicate across nodes and need a majority of replicas online to accept writes. A node that's down or partitioned doesn't necessarily break anything right away, which is exactly the danger. A three-node cluster running on two nodes is one failure away from every quorum queue rejecting publishes, and it'll run like that for weeks if nothing checks.

How to detect it. Cluster membership and running nodes:

rabbitmqctl cluster_status

Then the check that matters before any maintenance or after any node event:

rabbitmq-queues check_if_node_is_quorum_critical

It fails if shutting down this node would cost any quorum queue its majority. For a single queue, rabbitmq-queues quorum_status <queue-name> shows the leader and each member's state. In the management UI, Queues lists quorum queue members and flags queues with offline replicas.

Typical impact. Lost redundancy is invisible until the second failure. That second failure is a hard write outage on every affected queue, and a much messier recovery than restarting one node would have been.

6. Connection and channel churn

What it is. AMQP connections and channels are meant to be long-lived. An application that opens a connection per message (usually a framework misconfiguration or a missing connection pool) hammers the broker with constant TCP setup, TLS handshakes and authentication. RabbitMQ calls this high connection churn, and it degrades the whole node, not just the offending app.

How to detect it. The management API's overview endpoint tracks churn rates directly:

curl -s -u monitoring:password http://localhost:15672/api/overview | \
  jq '.churn_rates'

Compare connection_created_details.rate with connection_closed_details.rate. Healthy apps show near zero on both after startup. Sustained double-digit creation rates mean some client is cycling connections. rabbitmqctl list_connections name connected_at channels (or the Connections tab sorted by age) points at the culprit: hundreds of connections a few seconds old, all from one service.

Typical impact. Higher broker CPU, exhausted file descriptors, and latency for every well-behaved application on the node. It also floods the logs, burying signals from problems 1 to 5.

Reading a backlog: one worked example

The six checks pay off when you run them in order against one symptom. Say the orders queue is at 400,000 messages and climbing. There are four plausible readings, and each predicts something different:

Burst. Publish rate spiked, deliver rate is normal. The rate check from signal 1 shows publish well above its usual level and deliver steady.
Dead consumer. Deliver rate fell to zero. list_queues shows consumers at 0.
Stuck consumer. Consumers are connected but not acking. list_channels shows messages_unacknowledged pinned at prefetch_count.
Poison loop. Consumers are rejecting and the messages come back. The DLQ (or the redelivery rate, if there's no DLX) grows in step with the backlog.

Run the checks in that order and you'll usually rule out three readings within a few minutes. Say consumers reads 6, deliver rate is near zero, and two channels sit at exactly 250 unacked with a prefetch of 250: reading 3. Then check rabbitmq-diagnostics alarms before restarting anything. If a memory alarm is already active, the next thing to break is every publisher in the system, and that changes what you fix first.

What healthy looks like

Signal	Healthy	Worth a look	Page someone
Depth vs drain	Publish and deliver track each other	Depth up for an hour, rates close	Deliver near zero, depth climbing
Unacked	Low and moving	Rising on one channel	Pinned at prefetch, not moving
DLQ	Empty, or drained by a consumer	Nonzero for over a day	Growing every hour
Alarms	None	Node within 10% of watermark	Alarm active, publishers blocked
Quorum	All members online	One member lagging	Node quorum-critical
Churn	Near zero after startup	Steady single digits per second	Sustained double digits from one client

The thresholds in the middle column are starting points, not RabbitMQ defaults. Tune them to your traffic.

Why the dashboard alone doesn't save you

The management UI shows all six signals. It showed them during the three-day incident in the opening too. The data was there the whole time. These failures develop over hours and days, in metrics nobody stares at, and the pattern that matters (rate divergence, a flat unacked count, a slowly filling DLQ) only shows up as a trend, never in a snapshot.

The fix is making these checks systematic. Part two of this series builds that with native tools only: rabbitmqctl one-liners, management API queries you can cron, the built-in Prometheus plugin, and the policies that act as guardrails.

Watching these signals continuously

Everything above can also run without someone remembering to run it. CloudThinker connects to your RabbitMQ management API read-only and tracks all six signals as trends: depth versus drain, unacked buildup, DLQ growth, alarms, quorum health and churn. When one degrades, it runs the worked example above, names the cause, and proposes a fix that waits for your approval. Part three shows how.