Connect PagerDuty, Get an Alert Noise Audit, Then Watch MTTR Fall

It's 3:12 a.m. and the phone goes off: disk-usage-warning on the batch fleet, again. The on-call acknowledges from bed, glances at the graph, sees the usual sawtooth and goes back to sleep. Forty minutes later the incident resolves on its own. Nobody writes anything down, because nothing happened. The same alert fired 411 other times last month.

This is part five of our Connection Value series. The opener laid out the ladder: every connection should pay for itself in the first hour, then keep climbing. This post is the incident-tooling rung. It uses PagerDuty as the example, though everything here applies equally to Better Stack, Opsgenie and the other alert sources among CloudThinker's 42+ connections.

The promise is concrete. Within the first hour of connecting, you get an Alert Hygiene report on your last 30 days of paging history. Within the first weeks, real pages start arriving with a root-cause hypothesis already attached. And every month after, you get an MTTD, MTTA and MTTR view that shows the delta since you turned this on. Measured, not promised.

---

Why noisy alerts survive

Every team agrees alert noise is bad. It persists anyway, for structural reasons:

Alerts are added on incidents and never removed. A postmortem ends with "we should have been paged for this," so someone adds an alert. No postmortem ends with "this alert paged us 400 times for nothing," because nobody holds a postmortem for a page that self-resolved.

Thresholds are set on launch day. An 80% disk threshold on a batch node was sensible when jobs were small. Now the fleet runs at 78% most of the time, and every nightly run crosses the line and drops back.

Acknowledged looks like handled. In the incident log, a page that was acknowledged and then auto-resolved looks almost identical to one that someone fixed. Without reading the log entries, you can't tell them apart.

Nobody sees the aggregate. Each page is judged alone, by one tired person, at night. The 30-day picture, where two sources produce most of the volume and almost none of the action, never appears on any screen by default.

01Origin

Added, never removed

Every postmortem adds an alert. No process ever retires one.

02Threshold

Set on launch day

A limit that made sense for the first month now sits inside normal load.

03Response

Ack and close

A page that was acknowledged looks handled, even when nobody did anything.

04Visibility

No 30-day view

Each page is judged alone at 3 a.m. Nobody sees the aggregate.

The mechanism behind alert fatigue is covered in depth in our alert fatigue and triage post. What follows is how to measure it on your own account.

---

What the audit looks like by hand

You can build a rough version of the report yourself with the PagerDuty REST API. Start by pulling resolved incidents for a 30-day window:

curl -s -G "https://api.pagerduty.com/incidents" \
  -H "Authorization: Token token=$PD_API_KEY" \
  -H "Accept: application/vnd.pagerduty+json;version=2" \
  --data-urlencode "since=2026-09-01T00:00:00Z" \
  --data-urlencode "until=2026-10-01T00:00:00Z" \
  --data-urlencode "statuses[]=resolved" \
  --data-urlencode "limit=100" \
  | jq -r '.incidents[] | [.service.summary, .title, .created_at, .last_status_change_at] | @tsv'

The endpoint pages with offset and returns more: true while there's data left, so a real export loops until more is false. Group the output by service and title and you have volume.

Volume is the easy half. Whether a page was actionable lives in each incident's log entries:

curl -s "https://api.pagerduty.com/incidents/$INCIDENT_ID/log_entries" \
  -H "Authorization: Token token=$PD_API_KEY" \
  -H "Accept: application/vnd.pagerduty+json;version=2" \
  | jq -r '.log_entries[] | [.type, .agent.type, .created_at] | @tsv'

A resolve_log_entry whose agent is a service (the monitoring integration closing the incident) rather than a user, with nothing but an acknowledgement in between, is a page where no human did anything. Run that for every incident, join it back to the volume table, and rank. It works. It's also a day of scripting that nobody does twice, which is why the 30-day view rarely exists.

---

The first hour: an Alert Hygiene report

01Alert source

Paging tool

• PagerDutyIncidents, log entries, services, schedules
• Better StackIncidents and on-call
• OpsgenieAlerts and schedules

Read-only API key

02Investigation

CloudThinker

• Audit30 days of pages, ranked by noise
• InvestigateMetrics, logs, topology, recent deploys
• HypothesizeCause, evidence, confidence
• ReconstructTimeline and postmortem draft

Acknowledges and resolves nothing

03Where the team works

Chat and tickets

• Slack or TeamsHypothesis in the incident channel
• JiraPostmortem draft with evidence
• OnCall roomA shared room for serious incidents

Same rotation, better input

The connection itself is a read-only API key. The connection guide lists the exact scopes, so your security review can see what's granted and what isn't. At connection time the agent can read alerts, incidents, services and schedules. It can't acknowledge, resolve, reroute or silence anything.

Once connected, CloudThinker Resolve pulls your last 30 days of alert history and renders an Alert Hygiene report:

• Total alert volume, broken down by service and severity
• The percentage that was actually actionable: alerts where a human did something other than acknowledge and close
• Your top noise sources, ranked by pages generated per action taken
• Concrete deduplication and routing recommendations, per source

Here's an illustrative excerpt, a composite of what a first report tends to look like for a team paging on a few dozen services. Your numbers will differ:

The pattern this table usually reveals: a small number of sources generate most of the pages and almost none of the action. In the composite above, two sources account for roughly 70% of volume and 1% of the actions. That's not an unusual ratio. The report gives you a prioritized cleanup list on day one, from history you already have. Nothing needs write access to produce it.

---

One noisy source, read end to end

A row in the report is a conclusion. Here's how the agent gets to one, using the top row.

1. Name the readings. disk-usage-warning paged 412 times. There are a few explanations. Disks are genuinely filling, and the pages are early warnings of a real capacity problem. The threshold sits inside normal operating range, so ordinary jobs cross it. Or one host is flapping and generating most of the volume. Each predicts different data.

2. Check what each predicts. A real capacity trend would show usage climbing week over week and pages followed by human action (a resize, a cleanup ticket). A tight threshold would show pages clustered around batch windows, resolving on their own as jobs finish. A single flapping host would show one hostname in most incident titles.

3. Look at what came back. The log entries show 96% of these incidents resolved by the integration within an hour, with an acknowledgement and nothing else. The created timestamps cluster between 01:00 and 03:00, when the nightly batch runs. The pages are spread across the fleet, not one host. Disk usage, pulled from the monitoring connection, peaks near 85% and returns to 70% every morning with no upward trend.

4. Recommend at the source. That's a threshold sitting inside the batch peak, not a capacity problem and not a single bad host. The recommendation is to raise the threshold to 90% and keep the page, so a real fill-up still wakes someone. Muting the alert would have hidden the next genuine incident on the fleet.

That last step matters. The audit recommends fixes to the alert definition, so the noise goes away for a reason someone can review.

---

The ladder up: investigation on page

Cleaning up noise is the first-hour win. The ongoing value is what happens when a real alert fires.

01Signal

Page fires

An incident opens on a service and notifies the on-call.

02Orient

Investigate

Metrics, logs, blast radius and the nearest deploy, in parallel.

03Explain

Hypothesis

A cited cause with a confidence level, posted to the channel.

04Decide

Human verifies

The on-call checks the evidence instead of gathering it.

05Close

Timeline

Every event and action, reconstructed into a postmortem draft.

With the connection in place, the agent treats every page as an investigation trigger. Before the on-call engineer has finished reading the notification, it has already:

1. Pulled the relevant metrics for the alerting service and its immediate dependencies
2. Sampled the logs around the alert window for error-rate changes and new exception types
3. Mapped the blast radius from your topology: what's upstream, what's downstream, what else is degrading
4. Found the nearest deploy. If you've also connected a repo, it checks what shipped to the affected services in the hours before the alert and diffs the candidate change

It then posts a root-cause hypothesis to the incident channel: what's failing, the correlated deploy or metric inflection, the evidence and a confidence level. It's a hypothesis with citations, so the human can verify rather than trust.

The number to watch is alert-to-hypothesis time: the gap between the page firing and the hypothesis landing in the channel. The target is under five minutes, and it's displayed per incident in the dashboard. Some incidents beat it comfortably. A genuinely novel failure mode won't, and the timestamp will say so. Either way you can see what you're getting.

Why this moves MTTR: for most teams, the longest stretch of an incident isn't the fix. It's orientation, the twenty to forty minutes of "what's actually broken and what changed?" We broke that down in how to reduce MTTR. A cited hypothesis at minute four goes straight at that stretch.

---

After resolution: the timeline writes itself

When the incident closes, the agent reconstructs the timeline from the sources it was already watching: first alert, related deploys, metric inflections, escalations and every action taken, by a human or the agent. That reconstruction exports straight into a postmortem draft. If you've connected Jira, it joins the same post-resolution loop: the draft becomes a ticket with the evidence attached, and recurring incident patterns start showing up in your toil analysis.

Then, monthly, the before and after: an MTTD, MTTA and MTTR view with the delta since CloudThinker was enabled, per service and overall. This is the honest scoreboard. If dedup recommendations cut your page volume 60% but MTTR hasn't moved, you'll see that too. The usual reason is that the remaining alerts are the hard ones, which is where the investigation-on-page loop earns its keep over the following months.

---

Why this beats the traditional workflow

The deeper difference is where the judgment sits. PagerDuty is excellent at getting the right person's phone to ring. What it can't do is look at the evidence after the ring and decide what it means, and that's where the on-call's time actually goes. The connection takes on the looking, so the person spends their time on the deciding.

---

Gated by design

Everything above works with read access plus permission to post messages. Whether the agent may ever act on an incident is a separate, deliberate decision. CloudThinker acts within Auto Mode, so you decide what runs on its own and what waits for approval.

On SEV-1 and SEV-2, most teams keep it to investigate and escalate: the agent posts its hypothesis and pages per your on-call schedule, and touches nothing. Remediations for known patterns (restart the wedged consumer, recycle the connection pool) are portable SKILL.md files, and every one dry-runs in CloudThinker's Sandbox before it can touch a real system.

Escalation reads your on-call schedule, so "escalate" means paging the right human at 3 a.m., not posting into a channel nobody's watching. Every action lands in the audit trail: each hypothesis posted, each approval requested, who approved and what changed, with RBAC controlling who can change what the agent may do. The runbook automation post covers how teams decide which incident classes to automate.

---

What the agent won't do

"AI agent connected to your pager" deserves scrutiny, so here are the limits in plain terms:

• On high-severity incidents, the default is investigate and escalate. The agent hands a hypothesis to the on-call human and gets out of the way. It doesn't play hero on a SEV-1 unless you've explicitly allowed it, and most teams never do, on purpose.
• Read-only access reads; it doesn't touch. With the default connection, the agent can't acknowledge, resolve, silence or reroute a single alert. The Alert Hygiene recommendations are yours to apply. It won't change a routing rule for you without write access and your approval.
• Read-only also has limits on what it can see. The noise audit is only as good as your alert history. If a "resolved" alert was actually handled in a side channel PagerDuty never saw, the actionable percentage will undercount it. The report tells you what the data shows, and the data has edges.
• It doesn't skip the schedule. Escalation goes through your on-call rotation, not around it. Nobody gets paged, or bypassed, outside the policy you set.
• Nothing is invisible. Every hypothesis, proposal, approval and action is in the audit trail with a named human or your Auto Mode setting behind it.

The right mental model is a first responder who does the orientation work in the first five minutes, drafts the paperwork afterward, and only ever holds the tools you've handed over, one incident class at a time.

---

Getting started

Three steps. None of them needs write access.

1. Connect PagerDuty with a read-only key. The connection guide covers PagerDuty, Better Stack and Opsgenie, with the scopes for each.

2. Ask for the audit.

"Audit our PagerDuty incidents from the last 30 days. Rank alert sources by pages per action taken, show the actionable percentage for each, and recommend a threshold, dedup or routing change for the top five. Propose only, change nothing."

3. Point it at the next real page.

"When the next checkout incident opens, investigate before I get to it: pull metrics and logs for checkout and its dependencies, check what deployed in the last six hours, and post a hypothesis with evidence and a confidence level in the incident channel."

---

Related reading

• The connection value ladder
• Connect a repo, get your first PR review in minutes
• Connect Jira, get a toil analysis within the hour
• Alert fatigue and triage
• How to reduce MTTR
• Runbook automation
• Auto Mode

See your own noise numbers

The first-hour Alert Hygiene report needs nothing but a read-only connection and 30 days of history you already have. Most teams are surprised by their actionable percentage, usually in the wrong direction, and that surprise is where the MTTR curve starts to bend.

Try CloudThinker free (100 premium credits, no card required) and follow the connection guide to connect PagerDuty, Better Stack or Opsgenie and see your own noise audit within the hour.