PagerDuty Automation with AI Agents: From Page to Root Cause

Picture the incident note you'd actually want to read at 3 a.m. Not "high error rate on checkout-api". You already got that from the page. The note that says: payments-gateway is returning 503s, it started 4 minutes after deploy a1c9f2 shipped a connection-pool change, and the pool is now saturated at 100/100. That note names the failing dependency and the deploy that broke it, and it was written before you finished reaching for your laptop.

This last part of the series is about that note: PagerDuty automation that doesn't only route the page faster, but investigates the incident while you're still waking up. Part one mapped where on-call toil concentrates even on a well-run PagerDuty account: the context hunt after every ack, related pages arriving as separate incidents, escalations that fire on a responder who's busy rather than absent. Part two built everything native features give you: Event Orchestration for routing and suppression, alert grouping, service dependencies, Incident Workflows and Webhooks v3 for custom glue.

This part covers the layer none of that reaches. It isn't a PagerDuty replacement. Your services, escalation policies and schedules stay exactly where they are. CloudThinker sits on top of PagerDuty, investigates a triggered incident, and proposes the fix (or, under a policy you wrote, applies it), with your paging path untouched.

---

Why the page arrives empty

PagerDuty only knows what the alert told it: a summary, a severity, some custom details, a service. The evidence that explains the incident lives elsewhere: metrics and logs in your monitoring tool, pods and hosts in your cloud, deploys in your CI and Git history. Part two's whole native stack works on the alert stream, so even a perfectly tuned account delivers a well-routed, deduplicated incident with no diagnosis in it.

Writing the diagnosis in advance doesn't work either. A webhook runbook can check "was there a deploy in the last 30 minutes", but the next question depends on the answer. If yes, which service, and does the error start at that version? If no, is a dependency saturated, and which one? Every incident takes a different path through those questions, and no rule you wrote last quarter knows tonight's path.

---

Connecting PagerDuty: one read-only token, about five minutes

01Signal

PagerDuty

• New incidentsTriggered, escalated, acknowledged
• Incidents and log entriesUser API token, Observer role
• Services and dependenciesWhich service, which neighbors

Paging config untouched

02Investigation

CloudThinker

• Metrics and logsFrom the monitoring tools you connect
• Infrastructure statePods, hosts, cloud resources
• Recent changesDeploys and config in the window

No agents installed

03Where the team works

The incident

• Incident noteThe diagnosis is the first note
• Slack or TeamsApprovals and follow-up questions
• JiraFollow-ups with the evidence linked

Nothing new to watch

CloudThinker connects to PagerDuty the way your security review will want: a User API token owned by a user with read-only access.

You create the token under My Profile → User Settings → API Access → Create API User Token. A user token carries the permissions of the user who made it, so a token from an Observer can read incidents, services, schedules and escalation policies but change nothing. That's the point of using a user token rather than a general access REST key: the scope is whatever role you give that user. PagerDuty's docs on API access keys spell out the difference, so the review is a short read rather than a meeting.

No agents on your hosts, no changes to your PagerDuty configuration, no write access at connection time. An Observer token can't write incident notes, so until you move that user to Responder the diagnosis lands in your Slack or Teams channel instead. The PagerDuty connection guide has the step-by-step with the exact token type and roles.

PagerDuty tells CloudThinker which service and when. The evidence comes from the other tools you connect (your monitoring, your cloud or cluster, your source control), each with its own read-only credentials.

---

What happens when an incident triggers

01Trigger

Read the incident

Payload, service, custom details, and how often this signature fired.

02Evidence

Investigate

Metrics, errors and saturation for the service and its dependencies.

03Changes

Correlate

Deploys, config changes and flag flips in the window.

04Output

Incident note

Likely cause, confidence, proposed fix and rollback.

Once the incident triggers, CloudThinker runs the investigation an on-call engineer would run, in the first two or three minutes instead of the first half hour. Here's the checkout-api page from the opening, worked hypothesis first.

1. Read the incident context. The triggering event, which service and integration produced it, the payload and custom details, and how often this signature fired in the last 30 days, pulled from the same Incidents API part one used:

curl -s -H "Authorization: Token token=$PD_READONLY_KEY" \
  -H "Accept: application/vnd.pagerduty+json;version=2" \
  "https://api.pagerduty.com/incidents/$INCIDENT_ID/alerts" \
  | jq '.alerts[] | {summary, created_at, details: .body.details}'

A known-flappy signature gets flagged as such straight away. This one had never fired before, so it's treated as new.

2. Name the readings. A high error rate on checkout-api has three common explanations, and each predicts something different: its own code (errors on checkout-api alone, dependencies healthy), a saturated dependency (timeouts on calls to one downstream service, across every checkout-api instance), or a recent change (errors starting at a deploy or config timestamp).

3. Investigate across the infrastructure. Error rate, latency and saturation for checkout-api and its direct dependencies around the alert window, from whichever monitoring tool you connected. Here the errors were 503s from one call, payments-gateway, on every instance. That kills reading one. The gateway's connection pool showed 100 of 100 in use, which fits reading two, and raises the question of why the pool filled.

4. Correlate with recent change. Deploys, config changes, feature-flag flips and autoscaling events in the window. Most production incidents trace back to a change, and it's the check people do last and CloudThinker does early. Deploy a1c9f2 to payments-gateway landed four minutes before the first alert and changed the pool settings. Readings two and three turn out to be the same story.

5. Write the diagnosis where the responder looks. What fired, what the evidence shows, the most likely cause with a confidence level, and a proposed remediation (roll back a1c9f2) with a rollback note. When the responder opens the incident, the investigation is already there.

The difference from part two's orchestration is what gets automated. Orchestration follows rules you wrote in advance: route this, suppress that, group these. CloudThinker picks the next check based on what the last one found, the way an engineer reasons, and writes each step down as evidence.

---

Nothing runs outside the controls you set

CloudThinker investigates and posts the diagnosis. Writes like a rollback, a restart or a scale-out go through Auto Mode, so you decide what runs on its own and what waits for approval.

Your escalation policies stay exactly as configured. CloudThinker doesn't decide an incident isn't worth paging anyone for, and it never modifies a schedule, an escalation policy or an on-call assignment. If nobody acks, PagerDuty escalates on your timers, not CloudThinker's. What changes is that whoever picks up arrives at an investigated incident instead of a blank one. Every action lands in an audit trail: what triggered, what was checked, what was proposed, who approved, what changed and when.

---

What a first scan typically finds

Beyond live incident response, the first connection runs a hygiene pass over recent incident history: the part-one toil audit, done in minutes. The numbers below are illustrative, a composite of what a first scan tends to surface for a mid-market team running 40 to 80 services with a handful of on-call rotations. Yours will differ.

Most page volume comes from a handful of noisy signatures and un-grouped related incidents: the part-one fatigue story as concrete line items, and the fastest MTTR win on the board. And most real incidents in a composite like this correlate with a deploy, which is exactly the correlation that can be attached at trigger time instead of during the retro. Each row comes with a proposed fix.

---

Why this beats the traditional setup

The deeper difference is what each layer looks at. Everything native in PagerDuty acts on alerts and people. The investigation needs the infrastructure behind the alert, and that's the part an agent layer reads.

---

Prompts to try in your first session

CloudThinker is conversational. You ask in plain language, and the answer cites the PagerDuty and infrastructure data so you can verify rather than trust:

"The checkout-api high-error-rate incident just triggered. Read the alert payload, pull error rate and saturation for the service and its direct dependencies over the last hour, and check for deploys or config changes in the same window. Post the diagnosis to the incident and propose a fix, but change nothing else."

"Across the last 30 days, which alert signatures triggered the most incidents that auto-resolved without anyone acting, and what suppression or grouping would cut that noise without hiding a real signal?"

"For last night's payments outage, rebuild the timeline from trigger to resolve: the correlated deploy, which dependent services were affected, and every ack and escalation, as a postmortem draft."

Run the first one during your next real page. Compare the three-minute investigation, written before you're fully awake, with what you'd have assembled by hand, and you'll know quickly whether it earns a place in your response path.

---

What CloudThinker won't do

"AI agent connected to production incident response" should raise questions. The answers:

• Read-only by default. The connection uses a read-only token. Acting on infrastructure (restarting a service, rolling back a deploy, scaling a group) needs scoped write access and still goes through Auto Mode.
• No touching your paging. CloudThinker never edits escalation policies, schedules or on-call assignments. People stay paged.
• No silent changes. Every change shows up in chat with its evidence, whether it ran on its own or waited for approval.
• No unauditable actions. Every investigation, note, proposal, approval and change is in the audit trail with its evidence.

If you built the part-two automation, keep it. Event Orchestration and Incident Workflows are the right tool for paths that are already deterministic. Put CloudThinker on the part that never was: the investigation between "page delivered" and "cause found."

Closing the loop

Part one showed where on-call toil hides even on a healthy PagerDuty account. Part two showed how far native automation goes, and where it stops: orchestration decides who gets paged and when, and nobody investigates before a person arrives. That gap is what an investigation layer closes. Teams that add it on top of an existing PagerDuty setup typically bring MTTR down from tens of minutes to single digits on change-driven incidents, because the diagnosis is waiting in the incident instead of starting when someone opens a laptop.

Try CloudThinker free with 100 premium credits and no card required, and follow the PagerDuty connection guide to see your next triggered incident arrive already investigated.