Grafana Alerting Automation: AI Agents That Investigate Firing Alerts

A disk-usage rule on a production Kafka broker fires at 3:12 a.m. The threshold was tuned, the labels matched the right notification policy, and Grafana OnCall paged the right engineer within ninety seconds. The incident still ran forty minutes before anything useful happened. The engineer had to wake up, open the linked dashboard, abandon it for Explore, write a PromQL query to see whether growth was linear or a runaway, search Loki for broker errors in the same window, and scroll the annotations for the deploy that lined up. Grafana alerting automation, done by the book, routed the alert flawlessly. Delivery was never the bottleneck.

That forty-minute gap is what this final part of our Grafana alerting series closes. Part one mapped the dashboards-to-action gap and the short list of alerts that deserve a page. Part two built the full native stack (alert rules with clean labels, notification policies, mute timings, webhook contact points and OnCall escalation) and ended at its honest ceiling: routing is not investigation.

CloudThinker sits on top of your Grafana as the action layer. It receives firing alerts, queries the same data sources your dashboards read from, correlates evidence the way your on-call would, and changes things only within the scope you set. Nothing about your Grafana changes: the dashboards, the alert rules and your escalation chain stay where they are.

---

Why routing stops where investigation starts

Grafana is a window onto your data sources. The alert is the tip; the evidence lives underneath, in Prometheus, Loki, Tempo, CloudWatch or Elasticsearch, wherever the rule's query points. Native tooling routes the tip with real precision. Nothing in it follows the tip down.

The traditional ways of closing that gap all hit the same wall:

• Better runbooks tell the engineer what to check. They still do the checking, at 3 a.m., by hand.
• Webhook responders run the fix you scripted for the failure you predicted. A disk filling because of a new topic and a disk filling because of a stuck cleaner fire the same rule and need opposite fixes.
• More dashboards put more evidence on screen. Someone still has to read it, and the next question depends on what the last panel showed.

The work that's missing is branching: look at the evidence, decide what it rules out, choose the next query. That's what an engineer does in minutes one through forty, and it's the part that was never scriptable.

Connecting Grafana: one service account token, read-only

01Observability

Grafana

• Prometheus and MimirMetrics behind the rule
• Loki and TempoLogs and traces in the same window
• CloudWatch and othersAny data source the rule queries

Viewer service account

02Investigation

CloudThinker

• Pick up firing rulesRule, query, firing history
• InvestigateWider queries, logs, recent changes
• Propose the fixWith evidence and a rollback note

No plugins installed

03Where the team works

Chat and on-call

• Slack or TeamsFindings and approvals in the thread
• Grafana OnCall or IRMSame escalation chain, evidence attached
• PagerDutyIf that is what already wakes people

Nothing new to watch

The connection is a Grafana service account with the Viewer role, the same mechanism you'd use for any read-only API consumer. In Grafana 10 and later: Administration → Users and access → Service accounts → Add service account, set the role to Viewer, then Add service account token and copy it once. Paste the token and your Grafana URL into CloudThinker and the first scan starts. The flow is the same on OSS, Enterprise and Grafana Cloud, and takes about five minutes.

Because the role is Viewer, your security review is short. The token can read dashboards, alert rules, alert state and annotations, and query data sources through Grafana's own access controls. It can write nothing. Grafana's service account documentation spells out what each role grants. No plugin installs, no changes to your alert rules, no separate credentials for each data source: the agent queries through Grafana (POST /api/ds/query), so it sees precisely what your dashboards see and nothing more. The Grafana connection guide has the step-by-step.

When a rule fires: the investigation, not just the notification

01Signal

Rule fires

A Grafana-managed rule moves to Alerting. CloudThinker picks it up.

02Evidence

Investigate

Re-run the query wider and finer, pull logs, check the hosts underneath.

03Changes

Correlate

Deploy annotations, scaling events and config changes in the window.

04Output

Finding

What fired, what the evidence shows, likely cause, proposed fix.

When a rule transitions to firing, CloudThinker runs the sequence your on-call runs, in the first few minutes instead of minute forty:

1. Pull the rule's context. The query behind the rule, the condition that breached, its firing history over the past 30 days, and whether it's flapping between Alerting and Normal or in a genuinely new state.
2. Interrogate the data sources. Re-run the rule's query with a wider window and finer label breakdowns, pull error logs from the same time range, and check the health of the hosts or pods underneath, all through the data sources Grafana already trusts.
3. Correlate with change. Annotations, deploy markers, scaling events and config changes inside the window. Most production incidents trace back to something that changed. Humans check this last; the agent checks it first.
4. Deliver a finding, not a forward. What fired, what the evidence shows, the most likely cause, and a proposed remediation with a rollback note, posted to the same channel your contact point already targets.

The difference from the webhook pipeline in part two is branching. A webhook payload triggers steps you scripted in advance. The agent decides its next query based on what the last one returned.

---

The 3:12 a.m. Kafka alert, worked through

Here's the opening incident the way the agent works it. The details are illustrative, but the queries are the ones you'd run.

Three readings. A disk-usage alert on a Kafka broker usually means one of three things, and each predicts a different shape:

The first query. Free space and time-to-full, per broker:

predict_linear(node_filesystem_avail_bytes{job="kafka-node", mountpoint="/var/lib/kafka"}[6h], 4 * 3600)

Result: three of four brokers are flat. kafka-2 loses about 9 GB an hour and goes negative inside four hours. That rules out steady traffic growth, which would show on every broker.

The second query. Broker logs in the same window, through Loki:

{app="kafka", instance="kafka-2"} |~ "(?i)error|LogCleaner|Reassign"

No reassignment, no cleaner errors. That weakens the imbalance reading and leaves retention.

The change check. Deploy annotations show orders-service shipped at 22:40, and the broker log has a topic creation for orders-audit a minute later. Per-topic size from your Kafka exporter shows orders-audit growing on exactly the curve the disk is following, with its partitions concentrated on kafka-2. Its config: retention.ms=-1, set by the service's topic bootstrap.

The finding, posted in the alert thread at 3:16: disk on kafka-2 will fill in about four hours because the new orders-audit topic, created by the 22:40 deploy, has unlimited retention. Proposed fix, held for approval:

kafka-configs.sh --bootstrap-server kafka-2:9092 --alter \
  --entity-type topics --entity-name orders-audit \
  --add-config retention.ms=604800000

Rollback: --delete-config retention.ms to return to the previous value. Follow-up: a ticket for the orders-service team to set retention in their bootstrap, with the evidence linked.

The engineer reads one message instead of starting from a dashboard. Whether seven days is the right retention for an audit topic is a business question, which is why the change waits for a person.

---

The agent acts only as far as you allow

CloudThinker acts within Auto Mode, so you decide what runs on its own and what waits for approval, whether that's a silence, a rule edit, a restart or a config change like the retention fix above. Escalation stays intact either way. If the evidence says real outage rather than noise, the agent pages through Grafana OnCall, PagerDuty or whatever already wakes people, exactly as before, with the investigation attached to the page. Every investigation, proposal, approval and change lands in an audit trail.

What a first pass over a real Grafana typically finds

Alongside live alert response, connecting runs a hygiene review of the Grafana instance itself: the audit from parts one and two, compressed into minutes. The numbers below are illustrative, a composite of first scans on mid-market setups with 200 to 400 alert rules across Prometheus, Loki and a cloud data source. Yours will differ.

The shape matters more than the totals. Very little of this is "add more monitoring." Nearly all of it is existing monitoring that's quietly wrong: rules that can't fire, alerts that route nowhere, thresholds nobody has owned since the dashboard's author left. Each row arrives with a proposed fix that waits for approval.

Why this beats the traditional approach

The deeper difference is who holds the next question. In the native stack, every question after "which channel?" belongs to a person. With an agent on top, the person starts from an answer and decides whether to act on it.

Prompts to try in your first session

You talk to CloudThinker in plain language, and answers cite the underlying Grafana and data-source evidence so you can verify rather than trust:

"The disk-usage alert on the prod Kafka brokers just fired. Pull the rule's firing history, query the Prometheus data source for the six-hour trend per broker, check Loki for broker errors in the same window, and tell me whether this is steady growth or a leak. Propose only, change nothing."

"Which of our Grafana alert rules have been in NoData or Error state for more than two weeks, and in each case, is it the query, the data source or the target that broke?"

"Review the checkout team's alert rules: which ones flapped last month, and what threshold, pending-period or query changes would quiet them without losing real incidents?"

Run the first one during your next real incident and compare the agent's three-minute writeup with what your on-call assembled by hand. That comparison, not this article, is what should convince you.

What the agent won't do

An agent connected to production observability should raise questions. The answers:

• Read-only by default. A Viewer token can change nothing in Grafana. Acting (silencing an alert, editing a rule, restarting a workload) needs scoped write access you grant separately, and every change goes through Auto Mode.
• No silent changes. Every write follows the Auto Mode rules you set and shows up in chat.
• No bypassed escalation. The agent never decides an incident isn't worth waking someone for. Your OnCall chains and suppression rules stay yours, explicit and logged.
• Nothing outside the audit trail. Every query it ran, every conclusion it drew, every action it proposed or took is recorded with its evidence.

If you built the part-two stack, keep it. Notification policies and webhooks remain the right tool for deterministic routing. The agent takes the layer above: the investigating and deciding that routing was never going to do.

From routed to resolved

Part one showed why Grafana dashboards and alert rules drift away from what matters. Part two took native automation to its ceiling: routing is not investigation. The distance between "alert delivered" and "cause identified, fix proposed" is the forty minutes this article opened with, and it's the distance an agent layer closes. Teams that put investigation on top of a well-tuned Grafana typically cut time-to-diagnosis from tens of minutes to under five, and clear a quiet backlog of dead rules and broken routing in the first week as a side effect.

Try CloudThinker free with 100 premium credits and no card required, and follow the Grafana connection guide to see your next firing alert arrive already investigated.