Automated Root Cause Analysis with an AI SRE Agent: From Alert to Resolution

It's 02:47 and payment-api p99 latency has started to climb. Nothing has paged, because the threshold is p99 above 1s for five minutes and latency is at 400ms. If it keeps going, someone gets woken at 03:20, opens Datadog, then Grafana, then the CloudWatch console, then the deploy log, and around 04:00 works out that a release from two hours earlier leaks database connections. That's the hour automated root cause analysis is meant to take back.

This is Part 3 of our debugging trilogy, and the only one where we talk about our own product. In Part 1 you saw why root cause analysis takes hours: the evidence is scattered across five tools, and the correlation lives in one senior engineer's head. In Part 2 you built runbooks and wired your alerting stack together yourself. That work pays off, and it's also where the DIY approach hits a ceiling.

This article walks through how CloudThinker Resolve closes that gap: one realistic incident, end to end, from anomaly to verified recovery. If you want the vendor-neutral overview of RCA practice first, start with the root cause analysis guide.

---

Why runbooks stop where investigation starts

Runbooks are the right tool for the incidents you've already had. They break down for structural reasons, not because anyone wrote them badly.

• A runbook needs a human to start it. At 3 a.m. that human is the slowest step in the chain. The runbook doesn't run until someone wakes up, reads the alert, and picks the right page.
• Coverage is always behind. Runbooks exist for incidents you've seen. The one tonight is often new, or an old one in a new shape.
• Static thresholds see too late. The pager fires at a fixed line. A 2x latency shift on a payment path matters long before it crosses 1s.
• Correlation happens in someone's head. The deploy log, the database metrics and the traces sit in different tools. Joining them by timestamp is manual work, done under pressure, by whoever is on call.
• Investigation branches. Whether to check the database, the pool, the traffic or the node depends on what the last check showed. A runbook is a fixed list; real triage is a decision tree drawn on the fly.

Resolve takes the investigation itself: watching the right signals, joining them, checking each reading of the evidence, and handing a person a finding instead of an alert.

01Where the evidence lives

Your tools

• DatadogAPM traces, monitors, events
• PrometheusLatency histograms, service metrics
• GrafanaDashboards and alert rules
• CloudWatchDatabase and infrastructure metrics

Read-only connections

02Investigation

CloudThinker

• Anomaly detectionBaseline anomaly detection per service
• Topology graphBlast radius from live dependencies
• CorrelationDeploys, metrics and spans in one window
• RemediationRunbook match, rollback notes, verification

Resolve

03Where the team works

On-call and chat

• PagerDutyPage with the analysis attached
• SlackFinding, options and approval
• Audit trailEvery step, timestamped

Approval stays with a person

Resolve connects read-only to the tools you already use (Datadog, Grafana, Prometheus, PagerDuty, CloudWatch) and builds a live topology graph from them. Nothing is installed in your clusters, and nothing changes in your monitoring setup.

---

One incident, end to end

Everything below is a single illustrative incident: a payment API latency spike at a mid-market SaaS company. The numbers are plausible, not measurements. The point is the sequence, what an AI SRE agent does at each step and where a human stays in the loop.

01Detect

02:51

p99 at 2.2x its own baseline, far below the 1s pager threshold.

02Triage

02:51

Checkout sits on this path, so severity is high with zero alerts fired.

03Correlate

02:52

A 02:33 deploy, a saturating pool, latency inside database spans.

04Approve

02:58

Rollback prepared and paged to a person, who approves it.

05Verify

03:09

Latency back at baseline, pool at 40 percent, incident closed.

Detection: before the pager threshold

At 02:47, p99 latency on payment-api starts climbing, from a 180ms baseline to 400ms over four minutes. The PagerDuty threshold is p99 above 1s for five minutes, so nothing has paged.

Resolve's anomaly detection flags the deviation at 02:51. It isn't watching a static threshold. It's watching the metric against its own baseline for this service at this time of day. A 2.2x latency shift on a payment path is anomalous even though it's nowhere near the paging line.

Resolve opens an incident internally and starts working. No human has been woken yet, and depending on how this goes, none may need to be.

Triage: severity from blast radius, not alert count

The first thing a responder asks is "how bad is this?" Resolve answers from the live topology graph: the dependency map built from your connected tools, not a diagram someone drew last quarter.

payment-api sits upstream of checkout-web and downstream of orders-db, and checkout traffic flows through this exact path. That makes it high severity regardless of how many alerts have fired, which at 02:51 is still zero. Triage by blast radius means a quiet anomaly on a critical path outranks a noisy storm on a batch job.

Investigation: four readings, one survives

Latency on an API that talks to a database has a handful of usual causes, and each predicts different data. Resolve lays them out and checks each one rather than jumping to the most familiar.

01Reading 1

Database is slow

Predicts high CPU, IOPS or replication lag on orders-db. All at baseline.

02Reading 2

Pool exhausted

Predicts connections at the cap and time spent waiting in DB spans. Both true.

03Reading 3

Traffic surge

Predicts request rate up with latency. Request rate is flat.

04Reading 4

Noisy neighbor

Predicts CPU throttling on the node. None, and the API compute time is unchanged.

Is the database itself slow? CloudWatch shows orders-db CPU, read and write IOPS, and replication lag all at baseline. If the database were the problem, at least one would have moved.

Is the traffic different? Request rate on payment-api is flat across the window. More load would have shown up there first.

Where is the time going? Prometheus histograms split latency by span. If the API's own compute were slower, the non-database spans would grow. They didn't:

histogram_quantile(
  0.99,
  sum by (le, span_kind) (
    rate(span_duration_seconds_bucket{service="payment-api"}[5m])
  )
)

(Metric and label names depend on how you export spans. The shape of the query is what matters.) The added latency is all in span_kind="db".

Is the pool saturating? The connection pool in front of orders-db shows active connections climbing steadily from 02:35 and hitting the configured maximum of 100 at 02:49. Requests aren't waiting on the database. They're waiting for a connection to the database.

What changed? The deploy log has a release of payment-api at 02:33, two minutes before the connection count started to climb and 14 minutes before latency moved.

Put together: the 02:33 deploy changed connection handling, connections stopped being released, the pool saturated, and requests started queuing. The latency spike is the queue. A human would find the same three facts eventually, in three browser tabs, after about 40 minutes of digging. Resolve had them by 02:52.

Impact path: tracing it on the graph

Resolve traces the causal chain on the dependency graph: deploy on payment-api, then pool saturation on the payment-api to orders-db edge, then queuing latency on payment-api, then degraded checkout for users. It also confirms what's not affected. orders-db is healthy. The database is fine and the pool in front of it is exhausted. That one distinction saves the classic 3 a.m. detour of restarting a healthy database.

Remediation: propose, wait for a human, execute

Resolve matches the finding against your runbooks and proposes two options, each with impact and rollback notes:

This team approves production rollbacks by hand, so Resolve prepares option A in full (the exact command, the target revision, the verification checks) and pages the on-call engineer with the analysis attached. If payment-api runs on Kubernetes, the prepared action looks like this:

kubectl rollout history deployment/payment-api -n payments
kubectl rollout undo deployment/payment-api -n payments --to-revision=41
kubectl rollout status deployment/payment-api -n payments --timeout=10m

The engineer wakes at 02:58 to a diagnosed incident, not a raw alert. She reads the correlation summary, agrees, and approves from the page.

Resolve runs the rollback at 03:01. By 03:09, p99 is back at baseline and pool utilization has dropped to 40 percent. Resolve runs its verification checks, confirms recovery, and closes the incident. Verification matters as much as the fix: "rollback finished" and "service healthy" are different states, and the incident stays open until the second one is true.

The audit trail: the postmortem writes itself

Every step above landed in the audit trail as it happened: what Resolve found and when, which evidence it correlated, which readings it ruled out, what it proposed, who approved, which command ran, and what the verification showed. The postmortem timeline exists before anyone opens a doc. And because Resolve keeps episodic memory of past incidents, the next pool saturation on any service is recognized faster. The agent has seen this shape before.

Total time from anomaly to verified recovery in this illustrative run: 18 minutes, most of it waiting for a human to wake up and approve. Across incidents where teams let more actions run on their own, the platform has reported MTTR as low as 4m32s. That's a platform-reported figure from specific incidents, not a promise or a typical outcome.

---

Why this beats the traditional approach

The deeper difference is who carries the context. With runbooks, the context lives in the on-call engineer's head and has to be rebuilt every night from five tools. With Resolve, the context is assembled before anyone is paged, and the person's job becomes judging a finding rather than producing one.

---

You decide what runs on its own

The obvious objection to incident response automation: you don't want an agent running commands in production. Neither do we. Resolve acts within Auto Mode, so you decide what runs on its own and what waits for approval. Every action lands in the same audit trail, so letting more run on its own is a decision you make from evidence, not trust.

---

Try it yourself

Once your tools are connected, you can question an incident in plain language:

"What changed in the 30 minutes before the payment-api latency alert fired? Include deploys, config changes and scaling events."

"Show me the impact path for the payment-api latency anomaly, and tell me which services are not affected."

"Which runbook matches this incident, and what would rolling back payment-api affect? Propose only, change nothing."

These are the questions you'd ask a senior engineer at 3 a.m. The difference is that the answer arrives in seconds, with the supporting evidence linked.

What it never does without approval

Being specific about limits matters more than being impressive, so here's the list:

• Read-only by default. A fresh Resolve deployment observes, correlates and reports. Any write (a rollback, a restart, a pool change) needs a write grant, and still goes through Auto Mode.
• No emergency override. The agent never decides an incident is bad enough to skip the approvals you set.
• Everything is auditable. Findings, proposals, approvals, executions and verification results, all timestamped. If you can't reconstruct what the agent did and why, that's a bug, not a trade-off.
• Architectural judgment stays human. Resolve will tell you the pool saturated and roll back the deploy that caused it. Whether your service should hold database connections that way at all is a design decision, and it stays yours.

Where this sits next to Datadog's Bits AI

AI incident management isn't a one-vendor category. If you're all-in on Datadog, Bits AI is a useful investigation assistant, and it works well within the Datadog ecosystem. CloudThinker's difference is scope and execution. Resolve correlates and acts across your connected tools (Datadog, Grafana, Prometheus, PagerDuty, CloudWatch), because at most mid-market SaaS companies the evidence for one incident is spread across several of them. Beyond investigation, Resolve executes remediation under the approval model above. If your whole stack lives in one vendor, the calculus is different than if it doesn't.

Proof, and where to start

Amela, running a high-traffic consumer application on AWS, compressed incident response from hours to minutes with Resolve: root cause identified and a validated fix in under 30 minutes. And because Resolve runs on the same agent platform that optimizes your AWS costs, connecting your tools once covers more than incident response.

Start read-only: connect your monitoring stack, let Resolve observe for two weeks, and compare its findings against your own RCA. The free tier includes 100 premium credits, no card required.

Setup references: the incident response overview and connections guide cover wiring in Datadog, Grafana, Prometheus, PagerDuty and CloudWatch.

Related reading

• Part 1: Why root cause analysis takes hours
• Part 2: DIY incident response runbooks
• Root cause analysis guide
• Alert fatigue: a triage system that lets on-call sleep
• Auto Mode: decide what runs on its own