RabbitMQ Monitoring with Native Tools: rabbitmqctl, API, Prometheus

The most common RabbitMQ argument at 2 a.m. is "is this a traffic spike or a dead consumer?" Three columns from one command settle it:

rabbitmqctl list_queues name messages messages_unacknowledged consumers

01Burst

Outmatched

Depth climbs, consumers above zero, unacked near the prefetch ceiling and moving.

02Leak

Nobody home

Depth climbs and consumers read 0. Nothing is draining the queue at all.

03Stall

Stuck, not slow

Unacked pinned at prefetch times consumers, not moving, while depth grows.

If messages climbs while consumers is above zero and messages_unacknowledged hovers near your prefetch ceiling, consumers are alive and outmatched. That's a burst. If messages climbs while consumers reads 0, nothing is draining the queue. And if messages_unacknowledged sits pinned at exactly prefetch times consumer count while depth grows, consumers are receiving deliveries and never acking: stuck, not slow. That one line is the core of DIY RabbitMQ monitoring, and this guide builds the rest of the toolkit around it.

Part one of this series covered the six signals that matter: queue depth versus consumer throughput, unacked buildup, dead letter queues nobody drains, memory and disk alarms, cluster and quorum queue health, and connection churn. This part is the hands-on companion. It monitors all of them with only what ships in the box: rabbitmqctl, the management HTTP API, the built-in Prometheus plugin, policies and the alarm subsystem. Nothing to buy, nothing to install beyond plugins RabbitMQ already bundles.

01Triage

rabbitmqctl

Answers "what is happening right now" on the broker host in seconds.

02Script

Management API

The same data over HTTP, for jq sweeps, cron jobs and health checks.

03Alert

Prometheus

Time series and rules, so trends page someone instead of snapshots.

04Guard

Policies

DLX, length limits and TTL put a floor under how bad a queue can get.

---

Tool 1: rabbitmqctl, the triage layer

rabbitmqctl runs on the broker host (or anywhere with the Erlang cookie) and answers questions in seconds. Full reference in the rabbitmqctl man page.

Queue listings

The extended version of the opening command adds two useful columns:

rabbitmqctl list_queues name messages messages_ready \
  messages_unacknowledged consumers consumer_utilisation

How to read it. messages_ready is work waiting; messages_unacknowledged is work in flight. consumer_utilisation is the fraction of time the queue could deliver to a consumer immediately. A value well below 1.0 on a deep queue means consumers are the bottleneck (network, prefetch too low, or slow processing), not the broker. Run it twice, sixty seconds apart, and diff the messages column. Direction matters more than any absolute number.

Cluster status

rabbitmqctl cluster_status

Check three things in the output: every node you expect appears under Running Nodes, the Network Partitions section is empty, and the Alarms section is empty. For quorum queues, one more command tells you whether it's safe to restart a node:

rabbitmq-queues check_if_node_is_quorum_critical

If that fails, taking the node down would drop some quorum queue below its quorum. Reschedule the maintenance.

Connections, channels and consumers

rabbitmqctl list_connections name user state channels recv_oct send_oct
rabbitmqctl list_channels connection number consumer_count \
  prefetch_count messages_unacknowledged
rabbitmqctl list_consumers

How to read it. A connection in state blocked means a resource alarm is stopping its publishes (more on that in Tool 5). list_channels is where stuck consumers get names: a channel whose messages_unacknowledged equals its prefetch_count and never moves took deliveries and stopped acking. list_consumers maps queues to consumer tags and shows prefetch per consumer. A prefetch of 0 (unlimited) on a fast publisher is a memory incident waiting to happen.

Tool 2: the management API, the scriptable view

Everything the management UI shows is available over HTTP. Enable the plugin if you haven't (rabbitmq-plugins enable rabbitmq_management, port 15672), and create a dedicated user for scripts instead of reusing an admin account (management plugin docs):

rabbitmqctl add_user monitor 'use-a-real-password'
rabbitmqctl set_user_tags monitor monitoring
rabbitmqctl set_permissions -p / monitor "^$" "^$" ".*"

The monitoring tag gives the user the management API's node, connection and channel views. The three permission patterns are configure, write and read. One subtlety: RabbitMQ's read permission also covers consuming and purging over AMQP. If your scripts only need statistics, "^$" for read is enough; if they need to peek at dead-lettered messages, scope read to the DLQ names (for example "^.*\.dlq$") instead of ".*".

Top queues by depth

The /api/queues endpoint supports column selection, sorting and pagination server-side. Use them, because on a busy vhost the unfiltered payload is huge:

curl -s -u monitor:PASS \
  'http://localhost:15672/api/queues?columns=name,vhost,messages,messages_unacknowledged,consumers&sort=messages&sort_reverse=true&page=1&page_size=20'

Threshold sweeps with jq

Flag every queue past a depth threshold, and every queue with zero consumers that has messages waiting:

# Queues deeper than 10k messages
curl -s -u monitor:PASS http://localhost:15672/api/queues |
  jq -r '.[] | select(.messages > 10000)
    | "\(.vhost)/\(.name): \(.messages) msgs, \(.consumers) consumers"'

# Backed-up queues with no consumers at all
curl -s -u monitor:PASS http://localhost:15672/api/queues |
  jq -r '.[] | select(.consumers == 0 and .messages > 0)
    | "\(.vhost)/\(.name): \(.messages) stranded"'

The queue objects also carry rates: message_stats.publish_details.rate against message_stats.ack_details.rate is the in-versus-out comparison from part one, computed for you.

Node health and health checks

curl -s -u monitor:PASS http://localhost:15672/api/nodes |
  jq -r '.[] | "\(.name) fd=\(.fd_used)/\(.fd_total) disk_free=\(.disk_free) mem_alarm=\(.mem_alarm) disk_alarm=\(.disk_free_alarm)"'

Then the purpose-built health endpoints, which return HTTP 200 when healthy and 503 when not. They're ideal for load balancer checks and cron:

curl -s -u monitor:PASS http://localhost:15672/api/health/checks/alarms
curl -s -u monitor:PASS http://localhost:15672/api/health/checks/node-is-quorum-critical

Tool 3: the Prometheus plugin, metrics worth alerting on

The management API is fine for scripts. For real alerting you want the built-in Prometheus plugin:

rabbitmq-plugins enable rabbitmq_prometheus
curl -s localhost:15692/metrics | head -50

By default, queue metrics are aggregated per node. For per-queue alerting either scrape /metrics/per-object or set prometheus.return_per_object_metrics = true in rabbitmq.conf. The usual caveat applies: per-object mode gets expensive past a few thousand queues.

The short list worth paging on:

Three alert rules to start from, in Prometheus rules-file format:

groups:
  - name: rabbitmq
    rules:
      - alert: RabbitDeadLetterQueueNotEmpty
        expr: sum by (queue) (rabbitmq_queue_messages_ready{queue=~".*(dlq|dead).*"}) > 0
        for: 1h
      - alert: RabbitQueueGrowing
        expr: delta(rabbitmq_queue_messages_ready[15m]) > 1000
        for: 15m
      - alert: RabbitResourceAlarm
        expr: max(rabbitmq_alarms_memory_used_watermark) > 0 or max(rabbitmq_alarms_free_disk_space_watermark) > 0

The first one matters more than it looks. A dead letter queue is only useful if its depth is an alert, not a landfill.

Tool 4: policies, DLX, TTL and length limits as guardrails

Monitoring tells you a queue is filling. Policies put a floor under how bad it can get. Three are worth setting on day one.

Dead-lettering everywhere. Rejected and expired messages should land somewhere you can inspect, not vanish (DLX docs):

rabbitmqctl set_policy dlx-prod "^prod\." \
  '{"dead-letter-exchange":"dlx"}' --apply-to queues

One trap: a dead letter exchange with no queue bound to it drops dead-lettered messages silently. Bind a catch-all DLQ, then wire it into the first Prometheus rule above.

Length limits with explicit overflow behavior (max length docs):

rabbitmqctl set_policy cap-events "^events\." \
  '{"max-length":100000,"overflow":"reject-publish"}' --apply-to queues

reject-publish pushes backpressure to producers. The default, drop-head, discards your oldest messages (or dead-letters them, if a DLX is set). Pick deliberately. For most work queues, losing the oldest jobs unnoticed is worse than a publish error.

TTL on queues holding disposable data (TTL docs):

rabbitmqctl set_policy ttl-cache "^cache\." \
  '{"message-ttl":3600000}' --apply-to queues

One more trap: only one policy applies to a queue at a time, the one with the highest priority. Three policies whose patterns overlap don't stack. If you want DLX plus TTL on the same queues, put both keys in one policy definition. With a DLX in that definition, TTL expiry routes to the DLQ instead of deleting, which is exactly what you want for post-incident analysis.

Tool 5: reading memory and disk alarms

When RabbitMQ crosses its memory watermark or free disk floor, it blocks all publishing connections cluster-wide while consumers keep working (alarms docs). It's a self-protection mechanism, and it's the single most misdiagnosed RabbitMQ failure mode: producers hang, application timeouts fire, and nothing in the broker logs looks like an error.

Check for active alarms and see where memory is actually going:

rabbitmq-diagnostics alarms
rabbitmq-diagnostics check_local_alarms
rabbitmq-diagnostics memory_breakdown --unit mb

If memory_breakdown shows queue processes dominating, the memory alarm is a queue-depth problem in disguise. Fix the consumers, not the watermark. Then review the thresholds themselves in rabbitmq.conf:

vm_memory_high_watermark.relative = 0.5
disk_free_limit.absolute = 5GB

The default disk free limit is a token 50MB. That's far too low for production, because by the time it fires the broker has almost no room left to page messages to disk. Set it to at least a few GB, and alert on rabbitmq_disk_space_available_bytes well before the limit.

---

Which tool answers which question

The right-hand column is where every on-call shift actually goes. Native tooling is very good at what and where. It has nothing to say about why.

Where DIY thresholds stop

Run everything above and you have solid coverage: depth, unacked, consumers, DLQs, alarms, cluster health. You should build it. You should also be honest about the three walls you'll hit.

01Native tools

Notice

A rule fires: 40,000 ready messages on prod.payments, zero consumers.

02Native tools

Guard

A policy caps the queue and routes rejects to a DLQ you can inspect.

03Still a person

Explain

Which consumer stopped, why, and whether the deploy 20 minutes ago did it.

Thresholds notice depth. They don't explain it. An alert tells you prod.payments has 40,000 ready messages and zero consumers. It doesn't tell you which consumer group stopped, whether it crashed or deadlocked, or that the deploy 20 minutes ago changed a serialization format. Someone still SSHes in at 2 a.m. and runs the Tool 1 commands by hand.

DLQ contents don't analyze themselves. Knowing the dead letter queue has 3,000 messages is one alert. Understanding that 92% of them share a payload pattern a schema change broke is an hour of manual message inspection, reading x-death headers one message at a time.

The scripts are now a system you own. The jq sweeps, the PromQL, the cron'd health checks all need maintenance, tuning against alert fatigue, and an on-call rotation that remembers what each one means.

What comes next

Everything in this guide can run continuously, with the investigation attached. CloudThinker connects to the RabbitMQ management API read-only, watches depth trends, unacked buildup, DLQ growth and node alarms around the clock, and when something degrades it works out which consumer stalled and why before proposing a fix for your approval. That's part three of this series.

Related reading

• RabbitMQ monitoring: 6 signals that predict an outage
• Automating RabbitMQ monitoring and operations with AI agents
• How to triage Kafka consumer lag with native tools

Or see it against your own broker: try CloudThinker free with 100 premium credits and no card required.