Automate Prometheus Alerting with AI Agents: From Page to Root Cause

Ask any on-call engineer what happens after a Prometheus page and you'll get the same answer: a ritual of about ten PromQL queries, typed half-awake, in roughly the same order every time. Is the target even up? Is the error rate real or one bad instance? What does the latency histogram say? Did anything restart? Did anything deploy? The queries aren't hard. That's the point. Prometheus alerting tells you a symptom crossed a threshold, and the next thirty minutes of typing is how a person turns that symptom into a cause.

up{job="checkout-api"}
sum by (instance) (rate(http_requests_total{job="checkout-api",status=~"5.."}[5m]))
histogram_quantile(0.95, sum by (le) (rate(http_request_duration_seconds_bucket{job="checkout-api"}[5m])))
increase(kube_pod_container_status_restarts_total{namespace="prod"}[30m]) > 0
time() - process_start_time_seconds{job="checkout-api"} < 1800

This is part three of our Prometheus alerting series, and it covers what happens when those queries run themselves. Part one covered the rules side: symptom-based alerts, for: durations, the up == 0 blind spot, and the PromQL patterns behind alerts worth waking up for. Part two built the native automation ceiling: rule files, Alertmanager routing and inhibition, amtool silences, promtool tests. It ended on the honest limit that routing and silencing manage notifications. They don't investigate the alert.

CloudThinker is the layer that does. It doesn't replace Prometheus: your rules, your Alertmanager config and your Grafana dashboards stay exactly where they are. CloudThinker connects read-only, picks up firing alerts, runs the investigation your on-call would run, and proposes the fix (or, with your explicit permission, applies it).

---

Why the ritual never got automated

Part two's stack automates everything that's the same every time. The investigation isn't.

Each answer decides the next question. If up is 0, you're debugging scrape health, not latency. If one instance is the outlier, you look at that node; if all of them moved together, you look at a deploy or a dependency. A webhook handler can only encode the branches someone predicted.

The evidence is spread across jobs. The symptom lives in checkout-api. The cause might live in payments-api, in kube-state-metrics, or in a target that stopped reporting. Rules are scoped to one expression; the investigation crosses all of them.

Change data is checked last. Under pressure, people start with the graph that fired and get to "what deployed?" twenty minutes in. Most incidents trace back to a change, so the most productive question is usually the one asked latest.

Knowledge sits with whoever's been paged before. The ten-query ritual is written down nowhere. New on-calls rebuild it from scratch, one incident at a time.

---

Connecting: two read-only endpoints, about five minutes

01Monitoring

Prometheus

• Query API/api/v1/query, /api/v1/query_range
• Rules, alerts, targets/api/v1/rules, /alerts, /targets
• Alertmanager/api/v2/alerts, /api/v2/silences

Read paths only

02Investigation

CloudThinker

• Pick up firing alertsSkips what Alertmanager suppresses
• Test hypothesesScrape health, instances, changes, dependencies
• Propose the fixWith every query it ran attached

No exporters installed

03Where the team works

Chat and on-call

• Slack or TeamsFindings and approvals in the thread
• PagerDutySame escalation route, evidence attached
• GrafanaDashboards stay where they are

Nothing new to watch

CloudThinker talks to the same HTTP APIs you already use:

• Prometheus HTTP API. The base URL of your Prometheus server, or of a Thanos Query, Mimir or VictoriaMetrics frontend, since they speak the same API. CloudThinker uses GET /api/v1/query, /api/v1/query_range, /api/v1/alerts, /api/v1/rules and /api/v1/targets, all read paths.
• Alertmanager API (optional but recommended). CloudThinker reads GET /api/v2/alerts and /api/v2/silences to see grouping, inhibition and silence state, so it never investigates an alert your config has already suppressed.

You can check exactly what those calls return from your own terminal before connecting anything:

curl -s http://prometheus:9090/api/v1/alerts | jq '.data.alerts[] | select(.state=="firing") | .labels.alertname'
curl -s http://alertmanager:9093/api/v2/alerts | jq '.[] | {alert: .labels.alertname, state: .status.state}'

If your endpoints sit behind auth, you supply a bearer token or basic-auth credentials scoped however your reverse proxy allows. No exporters to install, no prometheus.yml changes, no remote-write, no admin API. Prometheus has no built-in granular authorization, which is exactly why the connection is designed to need nothing beyond query access. The Prometheus connection guide has the endpoint checklist your security review will want.

What happens when an alert fires

CloudThinker polls alert state continuously. When a rule transitions to firing and Alertmanager hasn't suppressed it, the investigation starts: the same one your ritual encodes, in the first two or three minutes instead of the first thirty.

01Signal

Alert fires

A rule moves to firing and Alertmanager has not suppressed it.

02Step zero

Trust the data

Check up and staleness on the targets the rule depends on.

03Hypotheses

Narrow it down

One instance or all, a deploy, a dependency. Each answer picks the next query.

04Output

Finding

Likely cause, the evidence, a proposed fix and its rollback.

1. Pull the alert's full context. The rule expression from /api/v1/rules, its for: duration, labels and annotations, how often this exact alert has fired in the past 30 days, and whether Alertmanager currently has it grouped with siblings or suppressed by an inhibition rule.
2. Check scrape health before trusting anything. up == 0 and staleness on the relevant targets, cross-checked against /api/v1/targets. Part one made the case that scrape failure is the silent lie in most Prometheus setups. "Is this data even current?" comes first, so you never chase a flat-lined graph that's really a dead exporter.
3. Correlate the symptom across related series. For a latency alert, that means the error-rate counters, saturation metrics and queue depths for the same job, plus the same metrics one hop upstream and downstream. It runs the rate(), histogram_quantile() and sum by (instance) comparisons a person would, and notices when one instance is the outlier versus all of them degrading together.
4. Compare against the last change window. Fresh process_start_time_seconds, container restart counters, image changes and rollout metrics from kube-state-metrics if present. People check this last; CloudThinker checks it early.
5. Deliver a finding, not a forward. What fired, what the evidence shows, the most likely cause named explicitly, and a proposed remediation with a rollback note. It's posted where your Alertmanager receiver already points, with every query it ran attached so you can re-run them yourself.

---

One alert, start to finish

Here's a single page walked through end to end. Service names, times and numbers are illustrative.

The signal. At 02:49, HighRequestLatency fires on checkout-api: p95 above 1 second for 10 minutes. Alertmanager hasn't grouped it with anything else, and there's no inhibition or silence on it.

Four readings. A latency alert like this usually means one of four things, and each predicts something different:

Bad data? No. All six targets report up = 1 and are scraping on schedule.

One instance? Also no:

histogram_quantile(0.95,
  sum by (instance, le) (rate(http_request_duration_seconds_bucket{job="checkout-api"}[5m]))
)

All six instances sit between 1.7 and 2.1 seconds, against a baseline around 210 ms. They moved together, which points away from a node problem and toward something they all share: the code, the config or a dependency.

A dependency? The obvious suspect is payments-api, which checkout-api calls on every order:

histogram_quantile(0.95,
  sum by (le) (rate(http_request_duration_seconds_bucket{job="payments-api"}[5m]))
)

Flat at its usual 90 ms. The slowness isn't coming from payments.

A change? This is where the evidence lines up:

count by (image) (kube_pod_container_info{namespace="prod", container="checkout-api"})

Every pod runs checkout-api:2.14.0, and process_start_time_seconds shows all six processes started between 02:31 and 02:36. The p95 graph steps up at 02:37, one minute after the last pod came up. Nothing else in the window changed.

The finding. "Latency regression on checkout-api began at 02:37, one minute after the 2.14.0 rollout completed. All instances affected equally; dependency latency and scrape health normal. Likely cause: the 2.14.0 release. Proposed action: roll back to 2.13.2 through your usual deploy path. Rollback note: re-deploying 2.14.0 restores the current state." Attached are the five queries above and their results.

The rollback waits for someone to approve it. The on-call wakes up to a cause, a proposal and the evidence, instead of a threshold and a blank expression browser.

---

Nothing runs outside the limits you set

CloudThinker acts within Auto Mode, so you decide what runs on its own and what waits for approval. Escalation stays intact either way. If the evidence says "real outage", CloudThinker pages through your existing Alertmanager route, faster and with the investigation already attached. It never decides an incident isn't worth waking someone for. Every investigation, proposal, approval and change lands in an audit trail with its evidence.

What a first pass typically finds

Alongside live alert response, the first connection runs a hygiene review of your rules and Alertmanager config: the part-one checklist, done in minutes. The numbers below are illustrative, a composite of a mid-market environment with a few hundred rules across two or three Prometheus servers. Yours will differ.

The shape mirrors what parts one and two predicted. Most rows aren't "add more alerting". They're "your existing alerting is noisy or misleading in specific, fixable ways". Each row arrives with a proposed fix gated by approval: a for: duration here, a recording rule there, an inhibit_rules block drafted for review.

Why this beats the traditional setup

The deeper difference is what each layer knows. Alertmanager knows labels: which alerts belong together and who should hear about them. It has never seen a PromQL result. CloudThinker starts where Alertmanager's knowledge stops, with the series behind the alert, and reasons from there.

Prompts to try in your first session

CloudThinker is conversational: plain language in, cited PromQL out, so you verify rather than trust.

"HighRequestLatency on checkout-api is firing. Investigate: check scrape health on the job first, compare p95 and error rate per instance for the last hour, and look for restarts or fresh process start times in the same window. Propose only, change nothing."

"Which of our alerting rules fired more than five times last week without anyone acknowledging them, and what for: duration or threshold change would quiet each one without losing real signal?"

"Audit our scrape targets: which are down or flapping right now, and which jobs have no up or absent() alert covering them at all?"

The first prompt is the one to run during your next real page. Put the three-minute writeup next to what your on-call assembled by hand, query by query, and you'll know whether it belongs in the escalation path.

What it won't do

A tool pointed at your monitoring stack should raise questions. The answers:

• Read-only by default. The connection needs query access only. Acting on the stack (posting a silence through the Alertmanager API, editing a rule file through your GitOps flow) needs you to grant that access, and every change goes through Auto Mode.
• No silent changes. Every write follows the Auto Mode rules you set and shows up in chat.
• No swallowed pages. Suppression logic stays yours, in your Alertmanager routes and inhibition rules, explicit and versioned. CloudThinker adds context to pages. It doesn't eat them.
• Nothing outside the audit trail. Every query run, finding posted and action taken is logged with its evidence and approver.

If you built the part-two setup, keep it. Routing, grouping, inhibition and tested rules are the deterministic layer, and they should stay deterministic. CloudThinker takes the layer that was never automatable before: reading the evidence and deciding what to check next.

Related reading

• Part one: 6 Prometheus alerting failure modes
• Part two: rules files, Alertmanager and promtool
• AI agents that investigate firing Grafana alerts
• Automated RCA with an AI SRE agent
• Auto Mode

The ritual, retired

Part one gave you rules worth firing. Part two gave you every native lever for managing the noise, and showed where the levers stop. What remains between "alert delivered" and "cause named" is the ten-query ritual, and that's what an investigation layer absorbs. Teams that put CloudThinker on top of an existing Prometheus and Alertmanager setup typically cut time to diagnosis from tens of minutes to under five, and fix the for:-less flappers and dead scrape targets in the first week along the way.

Try CloudThinker free with 100 premium credits and no card required, and follow the Prometheus connection guide to see your next firing alert arrive with the PromQL already run.