Automating RabbitMQ Monitoring and Operations with AI Agents

Two messages can land in your on-call channel at 14:50. The first says "queue depth over 100,000 on orders.process." The second says "the consumer group on orders.process stopped acknowledging at 14:32, eight minutes after the 2.4.1 deploy. The dead letter queue is filling with the same validation error from one producer, and node 2 is 10% away from a memory alarm that will block your publishers." Traditional RabbitMQ monitoring sends the first. Getting from the first to the second is the part that eats your evening.

This is part three of the RabbitMQ monitoring series. Part one mapped the signals that matter: queue depth versus consumer throughput, unacked buildup, dead letter queues, memory and disk alarms, cluster health. Part two built a DIY setup with rabbitmqctl, the management HTTP API and the Prometheus plugin, and ended on an honest limit: thresholds notice depth, but they can't tell you which consumer stopped and why.

This part closes that gap. CloudThinker sits on top of your existing RabbitMQ setup as an action layer. It watches the same signals continuously, investigates a degradation the way your on-call would, and proposes fixes that run only under the rules you set. Your broker, your Prometheus rules and your policies stay exactly where they are.

---

Why a threshold stops where the diagnosis starts

If you built the part-two setup, you have alerts on queue depth, unacked counts, DLQ length and node alarms. What you don't have is the next 45 minutes of work every one of those alerts triggers:

• Pull /api/queues and work out whether depth is growing because publish rate spiked (a burst) or because deliver and ack rates went to zero (a stalled consumer).
• List consumers on the affected queue, follow them through channels to connections, and find which service and host stopped acking.
• Peek at the dead letter queue, read the x-death headers, and work out whether messages were rejected, expired or overflowed, and whether the payloads share a pattern.
• Check /api/nodes for memory and disk alarms, because a blocked publisher looks exactly like a dead one from the application side.

None of these steps is hard. The problem is structural. A Prometheus rule evaluates one expression against one series; it has no idea that the queue it's watching is consumed by a service that deployed eight minutes ago. A runbook encodes one path through the investigation, and real backlogs don't follow one path. And the knowledge of which producer tends to ship malformed payloads lives in two people's heads. The person running these steps at 2 a.m. is rebuilding context the system already had.

---

Connecting RabbitMQ: read-only, about five minutes

01Broker

RabbitMQ

• Queues and rates/api/queues, message_stats
• Consumers and channels/api/consumers, /api/channels
• Nodes and alarms/api/nodes, /api/health/checks

monitoring tag, no write

02Investigation

CloudThinker

• Track trendsDepth vs drain, unacked, DLQ growth
• InvestigateConsumer, channel, connection, deploy
• Propose the fixWith evidence and rollback steps

Nothing installed on the broker

03Where the team works

Chat and on-call

• Slack or TeamsFindings and approvals in the thread
• PagerDutySame escalation path, diagnosis attached
• JiraProducer bugs filed with the DLQ evidence

Nothing new to watch

CloudThinker connects through the management API, the same HTTP interface you used with curl in part two. Create a dedicated user with the monitoring tag and no configure or write permissions:

rabbitmqctl add_user cloudthinker-monitor 'GENERATED_PASSWORD'
rabbitmqctl set_user_tags cloudthinker-monitor monitoring
rabbitmqctl set_permissions -p / cloudthinker-monitor "" "" ".*"

That gives visibility into queues, consumers, connections, channels and node health. The user can't publish, declare, delete or change policies. RabbitMQ's read permission does cover consuming and purging over AMQP, though, so if you want message inspection limited to dead letter queues, scope the read pattern to them (for example "^.*\.dlq$"), or use "" to keep the connection to statistics only. Repeat set_permissions for each vhost you want covered.

Paste the management endpoint URL and credentials into the connection screen, the first scan runs, and you're looking at findings in about five minutes. The RabbitMQ connection guide has the exact setup, including TLS and cluster-endpoint options, so your security review can see precisely what's granted.

What CloudThinker watches

Once connected, CloudThinker tracks the part-one signal set as trends with context, not point-in-time thresholds:

• Queue depth versus consumer throughput. Not "depth over N" but "depth growing while deliver and ack rates fall": the leak-versus-burst distinction a static threshold can't make.
• Unacked buildup. Rising messages_unacknowledged pinned to specific channels, which usually means a stuck consumer or a prefetch high enough to hoard messages one slow worker can't process.
• Dead letter queue growth. Depth and growth rate on every DLQ, plus the rejection reasons behind it.
• Memory and disk alarms. Fired alarms and near-misses. A node that reaches 90% of its high watermark every night during a batch window is a finding before it's an incident, because when the alarm fires RabbitMQ blocks every publishing connection cluster-wide.
• Cluster and quorum queue health. Node availability, partitions, and quorum queues running with fewer online replicas than they should have.
• Connection and channel churn. Services opening a connection per publish, which burns file descriptors and CPU across the cluster.

---

One backlog, investigated end to end

01Signal

Depth diverges

orders.process grows while its ack rate falls toward zero.

02Hypotheses

Investigate

Burst, dead consumer, stuck consumer, poison loop, blocked publishers.

03Changes

Correlate

When acks stopped, against deploys and producer versions in the window.

04Output

Finding

Which consumer, why, what the DLQ shows, and the proposed fix.

Here's the opening scenario, step by step (an illustrative example; service names and numbers are made up). orders.process crosses 100,000 messages at 14:50. Before touching any single explanation, CloudThinker lays out the readings and what each one predicts:

Alarms first. A memory alarm would explain everything downstream and change what to fix first. GET /api/nodes shows mem_alarm: false and disk_free_alarm: false on all three nodes, but node 2 is at 88% of its memory limit. Not the cause yet; noted for the finding.

Rates next. The queue's message_stats shows publish rate at its normal ~340/s. Deliver and ack rates dropped from ~340/s to near zero at 14:32. That rules out a burst. Something stopped draining.

Consumers. consumers reads 6, so they aren't dead. The channel view shows two channels at exactly 250 unacked with prefetch_count 250, unchanged for 18 minutes. The other four channels belong to the same service and show zero deliveries since 14:32. The connections behind them carry the client-provided name of the order-worker deployment.

Changes. The order-worker 2.4.1 rollout finished at 14:24, eight minutes before acks stopped.

DLQ. orders.dlq gained 2,100 messages since 14:30. CloudThinker peeks at a sample through the management API, requeueing so nothing is consumed:

curl -s -u cloudthinker-monitor:PASS -H 'content-type: application/json' \
  -X POST 'http://rabbit:15672/api/queues/%2f/orders.dlq/get' \
  -d '{"count":20,"ackmode":"ack_requeue_true","encoding":"auto","truncate":2000}'

Every sampled message carries an x-death header with reason: rejected and queue: orders.process, and every payload is missing the currency field that 2.4.1 started requiring.

Finding. Posted in the channel, with every API response attached: the 2.4.1 consumer rejects messages without currency, two workers are wedged on in-flight messages, the producer checkout-api 1.9 still sends the old shape. Proposed fixes: roll order-worker back to 2.4.0, then replay orders.dlq once a producer fix ships. Both wait for approval. Node 2's memory headroom is flagged as a separate follow-up.

Total time from threshold to finding: a few minutes, with the reasoning written down so you can check every step instead of trusting it.

What a first pass typically surfaces

The numbers below are illustrative, a composite of what a first scan tends to find on a mid-market cluster (three nodes, a few hundred queues). Yours will differ.

The shape is typical: one urgent item (the stalled consumer), one slow burn (the DLQ), and several structural risks nobody had noticed because they hadn't broken anything yet.

---

Proposals, not surprises

The finding above arrives with "roll order-worker back to 2.4.0" and "apply a DLX policy to the 41 uncovered queues" as proposals. CloudThinker acts within Auto Mode, so you decide what runs on its own and what waits for approval.

Every finding, proposal, approval and action lands in an audit trail: what was observed, what was proposed, who approved, what changed, and when. Remediation beyond RabbitMQ's own API (restarting a consumer service, rolling back a deploy) runs through your other connected systems under the same rules. The RabbitMQ connection itself stays read-only.

Why this beats the traditional setup

The deeper difference is where the investigation lives. With thresholds and runbooks it lives in a person, starting from zero each time the pager goes. With an agent on top, it starts the moment the signal degrades and arrives as a written chain of evidence the person can verify in a minute.

---

Getting started

Ask in plain language. Every answer cites the queue, consumer and node data behind it:

"Why is orders.process backing up? Consumers are connected but depth keeps climbing. Propose only, change nothing."

"What's in payments.dlq? Group the messages by rejection reason and tell me whether it's one bad producer."

"Is anything blocking publishers right now? Check memory and disk alarms on every node, and show me which node is closest to its watermark."

"Which queues have no dead-letter exchange, and what would a single policy covering them look like?"

Run the first one during your next real backlog and compare it with what your on-call would have assembled by hand.

What CloudThinker won't do

Software with access to your message broker should raise questions. Here are the answers:

• The connection is read-only. The monitoring user can't publish, declare, delete or reconfigure anything. Write actions need explicitly granted, scoped access and Auto Mode letting them run.
• No silent changes. No queue is purged, no policy applied and no consumer restarted outside what you've allowed.
• Nothing outside the audit trail. Every proposal and action is logged with its evidence and its approver.
• Architectural judgment stays yours. CloudThinker will tell you a service opens 1,100 connections an hour. Whether to refactor it to one long-lived connection with channels is your engineering call.

Related reading

• RabbitMQ monitoring: 6 signals that predict an outage
• RabbitMQ monitoring with native tools: rabbitmqctl, API, Prometheus
• Automating Kafka consumer lag response with AI agents
• Auto Mode: graduated autonomy with safe defaults

From watching queues to answering questions

Part one gave you the signals and part two gave you the commands. Neither gives you an investigation that starts the moment a signal degrades, at 2 p.m. or 2 a.m., and arrives as a diagnosis with an approval-gated fix instead of a number on a graph. That's the layer CloudThinker adds on top of the RabbitMQ you already run.

Try CloudThinker free with 100 premium credits and no card required, and follow the RabbitMQ connection guide to see your own cluster's first findings within the hour.