PagerDuty Automation: Where On-Call Toil Hides Even When Paging Works

The page fires at 02:14. The responder acks at 02:17 from their phone, sits up, opens a laptop, and starts the routine: which service is this, which Grafana board shows it, where do its logs live, did anyone deploy tonight. At 02:35 they can finally say what's broken. PagerDuty did its job perfectly. It got the right person out of bed inside three minutes. The other eighteen minutes were human toil, and they'll happen again on the next page.

Every incident has two timestamps you can read off the timeline. One is when PagerDuty paged someone. The other is when a person actually understood what was broken. The gap between them almost never shows up on a dashboard, and on most rotations it's the biggest chunk of your MTTR.

0102:14

Triggered

A monitor fires and PagerDuty opens the incident.

0202:17

Acknowledged

The responder acks from their phone and reaches for a laptop.

0302:18

Context hunt

Dashboards, logs, recent deploys, upstream health, stitched by hand.

0402:35

Understood

Someone can finally say what is broken and why.

Illustrative timeline. PagerDuty owns the first two timestamps. The gap before the last one rarely shows up on any dashboard.

PagerDuty is good at the first timestamp. Its automation gets the right person paged, escalates when they don't ack, and tracks who owns what. It doesn't close the gap to the second one. After the ack, a person still has to hunt for context, and that hunt is where PagerDuty automation stops and manual toil begins.

This guide walks the toil that survives even when your paging setup is clean. For each pattern: what it is, why it happens, how to measure it, and roughly what it costs your rotation. None of it is a knock on PagerDuty. It's a map of where the human minutes go once the page has done its job.

This is part one of a three-part series. Part two covers DIY incident automation with PagerDuty's native tools: Event Orchestration, Incident Workflows, webhooks and their ceiling. Part three shows automating investigation with an AI agent that goes from page to root cause before the engineer opens a laptop.

01After the ack

Context hunt

Ten to twenty minutes of setup before real diagnosis starts.

02Cascades

Split incidents

One cause, five incidents, several responders hunting in parallel.

03Escalation

Busy, not absent

Timeouts fire on a responder who is working but quiet.

04Afterwards

Hand-built timelines

Postmortems rebuilt from Slack scrollback and memory.

05Load

Pages per person

The burnout metric that nobody joins to the schedule.

Five places on-call toil survives a clean paging setup. Each section below has a way to measure it.

1. The context hunt after every ack

What it is. The responder acks, opens a laptop, and spends the next ten to twenty minutes rebuilding the situation from scratch. The alert said that something is wrong. It rarely says why, and never says what to look at first.

Why it happens. Alert payloads carry a summary, a severity and maybe a link to the monitor that fired. They don't carry the surrounding state: recent deploys, correlated metric movement, the health of upstream dependencies. That context lives in five other tools, and stitching it together is human work every time, including the third time this quarter the same alert fired.

How to measure it. The per-incident timeline has the raw numbers. Compare the created_at of the incident with the first substantive note or status update a responder posts (not the ack itself). That delta is your context-hunt tax. Pull a month of resolved incidents through the Incidents API:

curl -s -H "Authorization: Token token=$PD_TOKEN" \
  -H "Accept: application/vnd.pagerduty+json;version=2" \
  "https://api.pagerduty.com/incidents?since=2026-06-01T00:00:00Z&until=2026-07-01T00:00:00Z&time_zone=UTC&statuses[]=resolved&limit=100" \
  | jq '.incidents[] | {id, created_at, title}'

The endpoint pages at 100, so loop on offset while more is true. Then, for each incident, find the first note in its log:

curl -s -H "Authorization: Token token=$PD_TOKEN" \
  -H "Accept: application/vnd.pagerduty+json;version=2" \
  "https://api.pagerduty.com/incidents/$INCIDENT_ID/log_entries" \
  | jq '[.log_entries[] | select(.type == "annotate_log_entry")] | min_by(.created_at) | .created_at'

Typical impact. On a healthy rotation this is commonly 10 to 20 minutes per incident before real diagnosis starts. It's pure setup, repeated every page, and for most teams the single largest reducible slice of MTTR.

2. Related pages arriving as separate incidents

What it is. One root cause (a database failover, a bad deploy, a saturated node) trips five monitors. PagerDuty opens five incidents. Five pages go out, possibly to different responders, who each start their own context hunt on what's fundamentally the same event. Worse are the near-duplicates: the "high error rate" page and the "latency SLO burn" page tell the same story, and nothing ties them together unless you configured it to.

Why it happens. Deduplication keys collapse identical alerts from the same source. A cascading failure produces different alerts from different services. Without content-based grouping or mapped service dependencies, PagerDuty has no way to know that alert B is a symptom of alert A.

How to measure it. In the web app, open Analytics → Incidents and compare incident volume per service with the number of distinct causes you actually wrote postmortems for. A large ratio means fragmentation. Programmatically, list incidents in a tight window and cluster by service and time: bursts within a few minutes on related services are your duplicate suspects.

curl -s -H "Authorization: Token token=$PD_TOKEN" \
  -H "Accept: application/vnd.pagerduty+json;version=2" \
  "https://api.pagerduty.com/incidents?since=2026-06-01T00:00:00Z&until=2026-07-01T00:00:00Z&limit=100" \
  | jq -r '.incidents[] | "\(.created_at[0:16])\t\(.service.summary)\t\(.title)"' | sort

Read the sorted output for clusters that share a minute. Alert grouping settings live per service under Services → (your service) → Settings → Alert Grouping.

Typical impact. Fragmentation inflates page counts and pulls extra responders into a single event. Even one duplicate per real incident roughly doubles the human attention spent on that class of failure.

3. Escalations that fire because the first responder is heads-down

What it is. The escalation policy exists to catch a missed page: nobody acked, so it climbs to the next person. A common trigger is different. The first responder acked, is deep in logs, hasn't posted an update, and an escalation fires anyway (a re-escalation after the ack timeout, or someone escalating "just in case" because the incident looks quiet). Now a second engineer wakes up, acks, and starts their own context hunt on an incident that was already being worked.

Why it happens. Escalation measures acknowledgement and silence, not progress. An engineer investigating hard and saying nothing looks, to the policy and to everyone watching, identical to one who slept through it. The system optimizes for "someone is looking" and can't see that someone already is.

How to measure it. Count escalations recorded after the first ack. In the log entries, look for escalate_log_entry events later than the first acknowledge_log_entry:

curl -s -H "Authorization: Token token=$PD_TOKEN" \
  -H "Accept: application/vnd.pagerduty+json;version=2" \
  "https://api.pagerduty.com/incidents/$INCIDENT_ID/log_entries?is_overview=true" \
  | jq '[.log_entries[] | {type, created_at}]
        | (map(select(.type == "acknowledge_log_entry")) | min_by(.created_at) | .created_at) as $ack
        | map(select(.type == "escalate_log_entry" and $ack != null and .created_at > $ack))
        | length'

A cluster of non-zero results means escalations are firing on busy responders, not absent ones.

Typical impact. These are wake-ups with no diagnostic value. The second responder adds nothing the first wasn't already doing. They quietly drive alert fatigue, and they don't move MTTR at all.

4. Postmortem timelines rebuilt by hand from Slack

What it is. The incident resolves. Now someone has to write the postmortem, which means rebuilding the timeline: when it started, when we noticed, what we tried, what worked. Half of that lives in a Slack thread, half in the PagerDuty timeline, and the "what we tried" part lives only in the responder's memory. Building the narrative is an hour of archaeology, days after the fact.

Why it happens. The record is scattered. PagerDuty knows the paging and escalation events precisely. It doesn't know that someone rolled back a deploy in a terminal or ran a query that confirmed the cause. Those actions happened in tools that never wrote back to the incident, so a person has to remember and transcribe them.

How to measure it. PagerDuty's timeline is your skeleton. Pull it with the same log_entries endpoint and note how much of the finished postmortem came from it versus Slack scrollback and memory. If you use Postmortems in PagerDuty, the gap between the auto-populated timeline and your final narrative is exactly the manual reconstruction cost.

Typical impact. Commonly 30 to 90 minutes per postmortem of pure assembly, which is why postmortems slip, get thinner over time, or quietly stop happening for anything below SEV1.

5. Pages per on-call week that nobody is tracking

What it is. The metric that predicts burnout better than MTTR: how many times a person got paged during their on-call week, and how many of those pages landed on nights and weekends. Most teams can tell you last month's MTTR to the minute and have no idea that one engineer took 40 pages in seven days while another took four.

Why it happens. MTTR is the number leadership asks for, so it's the number that gets dashboards. Load per person means joining incident data to the on-call schedule (who actually held the pager when each page fired), and that join is nobody's job by default.

How to measure it. PagerDuty's analytics expose incident counts, but for true per-person load you correlate incidents with on-call shifts. Pull who was on call for a window:

curl -s -H "Authorization: Token token=$PD_TOKEN" \
  -H "Accept: application/vnd.pagerduty+json;version=2" \
  "https://api.pagerduty.com/oncalls?since=2026-06-01T00:00:00Z&until=2026-07-01T00:00:00Z&time_zone=UTC" \
  | jq '.oncalls[] | {user: .user.summary, schedule: .schedule.summary, start, end}'

Then bucket incidents into those shift windows, count per user, and flag the ones outside business hours. A common healthy target is under 10 actionable pages per on-call week, with off-hours pages the exception.

Typical impact. Uneven load and silent off-hours volume are leading indicators of attrition on an SRE team. They stay invisible until someone builds this join, usually right after a good engineer quits.

Why paging fixes don't touch this

All five patterns share a shape. PagerDuty does its job (routing, escalation, tracking) cleanly. The toil sits in the seam after the page and before resolution, where a person has to gather context, connect related failures, communicate progress, rebuild history and carry an uncounted load. None of that is a routing problem, so tuning escalation policies harder won't touch it.

It also isn't a one-time fix. Every new service adds another context hunt. Every deploy is a potential cascade of related pages. Every incident that resolves without writing back to the timeline is another hour of postmortem archaeology owed. The toil regrows at the rate you ship.

What a healthy rotation looks like

The first note on every incident says what's broken, posted within minutes of the ack.
One cause, one incident. Grouping and dependencies fold cascades before they page.
Escalation means absence, not silence. Responders post a short status early so timeouts don't fire on busy people.
Timelines write themselves from actions that report back to the incident.
Pages per person per week sit on the same dashboard as MTTR.

The honest first step is to measure: run the checks above for a month and put numbers on your context-hunt delta, your duplicate ratio and your pages per person. Then decide whether the fix is configuration (part two walks through what PagerDuty's native tools can automate) or whether the investigation itself needs to happen before the person arrives.

Close the gap to root cause

The two-timestamp gap is automatable. CloudThinker connects to PagerDuty with read-only access, picks up triggered incidents, and investigates the underlying infrastructure (metrics around the alert window, recent deploys, related service state), posting a diagnosis while the responder is still waking up (in the incident notes once you grant write access, in Slack or Teams until then). Your escalation policies stay exactly as configured, and people stay paged. Changes go through Auto Mode, so you decide what runs on its own and what waits for approval. Part three shows how it works end to end.

To start with your own numbers, ask something like this in your first session:

"For the last 30 days of PagerDuty incidents, how long did each one take from trigger to the first responder note, which incidents arrived within five minutes of each other on related services, and how many pages did each person take during their on-call shifts?"

Try CloudThinker free with 100 premium credits and no card required, or continue with part two.