Automated Root Cause Analysis with an AI SRE Agent: From Alert to Resolution

This is Part 3 of our debugging trilogy, and the only one where we talk about our own product. In Part 1 you saw why root cause analysis takes hours: the evidence is scattered across five tools, and correlation lives in one senior engineer's head. In Part 2 you built runbooks and wired your alerting stack together yourself. That work pays off — but automated root cause analysis is where the DIY approach hits a ceiling. Your runbooks still need a human to execute them at 3AM, and your coverage is never complete, because runbooks only exist for the incidents you have already had.

This article walks through how CloudThinker's Deep Response Engine (DRE) closes that gap: one realistic incident, end to end, from anomaly to verified recovery. If you want the vendor-neutral overview of RCA practice first, start with the root cause analysis guide.

One incident, end to end

Everything below is a single illustrative incident — a payment API latency spike at a mid-market SaaS company. The numbers are plausible, not measurements. The point is the sequence: what an AI SRE agent actually does at each step, and where a human stays in the loop.

Detection: before the pager threshold

At 02:47, p99 latency on payment-api starts climbing — from a 180ms baseline to 400ms over four minutes. Your PagerDuty threshold is set at p99 above 1s for 5 minutes, so nothing has paged yet.

Pulse, DRE's anomaly detection capability, flags the deviation at 02:51. It is not watching a static threshold; it is watching the metric's behavior against its own baseline for this service, at this time of day. A 2.2x latency shift on a payment path is anomalous even though it is nowhere near your paging threshold.

At this point DRE opens an incident internally and starts working. No human has been woken up yet — and depending on how this plays out, none may need to be.

Triage: severity from blast radius, not alert count

The first question a human responder asks is "how bad is this?" DRE answers it from the live topology graph — the dependency map built from your connected tools, not from a diagram someone drew last quarter.

payment-api sits upstream of checkout-web and downstream of orders-db. The graph shows that checkout conversion traffic flows through this exact path. That makes this a high-severity incident regardless of how many alerts have fired — which, at 02:51, is still zero. Triage by blast radius means a quiet anomaly on a critical path outranks a noisy alert storm on a batch job.

Correlation: the deploy you forgot about

Now the actual root cause work. DRE correlates events across every connected source — Datadog, Grafana, Prometheus, PagerDuty, CloudWatch — inside the anomaly window and the period before it. In this incident it finds three things that a human would eventually find, in three different browser tabs, after 40 minutes of digging:

1. A deploy event on payment-api at 02:33 — 14 minutes before the latency shift began.
2. A CloudWatch metric showing active connections on the orders-db connection pool climbing steadily from 02:35 and hitting the configured maximum of 100 at 02:49.
3. Prometheus histograms showing the added latency is concentrated in database call spans, not in the API's own compute time.

Correlated, these three facts tell one story: the 02:33 deploy changed connection handling, connections stopped being released, the pool saturated, and requests began queuing for a connection. The latency spike is the queue.

Impact path: tracing it on the graph

DRE traces the causal chain on the dependency graph: deploy on payment-api → connection-pool saturation on the payment-api to orders-db edge → queuing latency on payment-api → degraded checkout for users. It also confirms what is not affected: orders-db itself is healthy — CPU, IOPS, and replication lag are all at baseline. The database is fine; the pool in front of it is exhausted. That distinction alone saves the classic 3AM detour of restarting a healthy database.

Remediation: propose, wait for a human, execute

DRE matches the finding against available runbooks and proposes two options, each with impact and rollback notes:

This team runs DRE at the Approve level for production rollbacks, so DRE prepares option A — the exact rollback command, the target revision, the verification checks — and pages the on-call engineer with the full analysis attached. The engineer wakes up at 02:58 to a diagnosed incident, not a raw alert. She reads the correlation summary, agrees, and approves from the page.

DRE executes the rollback at 03:01. By 03:09, p99 latency is back at baseline and pool utilization has dropped to 40 percent. DRE runs its verification checks, confirms recovery, and closes the incident.

The audit trail: the postmortem writes itself

Every step above landed in the audit trail as it happened: what Pulse found and when, what evidence DRE correlated, which runbooks it proposed, who approved, what command ran, and what the verification checks showed. The timeline for the postmortem exists before anyone opens a doc. And because DRE keeps episodic memory of past incidents, the next connection-pool saturation on any service gets recognized faster — the agent has seen this shape before.

Total time from anomaly to verified recovery in this illustrative run: 18 minutes, most of it waiting for a human to wake up and approve. Across incidents where teams have granted more autonomy, the platform has reported MTTR as low as 4m32s — a platform-reported figure from specific incidents, not a promise or a typical outcome.

Graduated autonomy: four levels, per action, per environment

The obvious objection to incident response automation: you do not want an agent running commands in production. Neither do we, which is why nothing executes by default. DRE operates at one of four autonomy levels, configured per action class and per environment:

The granularity matters. A typical progression: everything starts at Notify in production. After a few weeks of reading DRE's findings and checking them against your own RCA, you move low-risk diagnostics to Suggest. Cache flushes and pod restarts in staging go Autonomous early. Production rollbacks might sit at Approve indefinitely — prepared and one click away, but always behind a named human. Every level's actions land in the same audit trail, so graduating an action class is a decision you make from evidence, not trust.

Try it yourself

Once your tools are connected, you can interrogate an incident in chat. Ask the agent:

• what changed in the 30 minutes before this alert fired?
• show me the impact path for the payment-api latency anomaly
• which runbook matches this incident, and what would rolling back affect?

These are the same questions you would ask a senior engineer at 3AM. The difference is the answer arrives in seconds, with the supporting evidence linked.

What it never does without approval

Being specific about limits matters more than being impressive, so here is the honest list:

• Read-only by default. A fresh DRE deployment observes, correlates, and reports. It executes nothing until you grant an autonomy level above Notify for a specific action class in a specific environment.
• No remediation without a grant. There is no "emergency override" where the agent decides an incident is bad enough to act on its own. If the action class is at Notify or Suggest, DRE does not touch your systems.
• Everything is auditable. Findings, proposals, approvals, executions, verification results — all of it, timestamped, in the audit trail. If you cannot reconstruct what the agent did and why, that is a bug, not a feature trade-off.
• Architectural judgment stays human. DRE will tell you the connection pool saturated and roll back the deploy that caused it. Whether your service should hold database connections that way at all is a design decision, and it stays yours.

Where this sits next to Datadog's Bits AI

AI incident management is not a one-vendor category. If you are all-in on Datadog, Bits AI is a genuinely useful investigation assistant, and it works well within the Datadog ecosystem. CloudThinker's difference is scope and execution: DRE correlates and acts across your connected tools — Datadog, Grafana, Prometheus, PagerDuty, CloudWatch — because at most mid-market SaaS companies the evidence for one incident is spread across several of them. And beyond investigation, DRE executes remediation under the approval model described above. Different problems, different tools; if your whole stack lives in one vendor, the calculus is different than if it does not.

Proof, and where to start

This is not hypothetical. Amela, running a high-traffic consumer application on AWS, compressed incident response from hours to minutes with DRE and Pulse — root cause identified and a validated fix in under 30 minutes. And because DRE runs on the same agent platform that optimizes your AWS costs, connecting your tools once covers more than incident response.

Start in Notify mode: connect your monitoring stack, let DRE observe for two weeks, and compare its findings against your own RCA. The free tier includes 100 premium credits, no card required.

Setup references: the incident response overview and connections guide cover wiring in Datadog, Grafana, Prometheus, PagerDuty, and CloudWatch.