PagerDuty Automation with Native Tools: Event Orchestration, Incident Workflows, and Webhooks v3

One team we know cut their weekly page count roughly in half in a single afternoon. They didn't buy anything. They wrote one Event Orchestration rule that recognized a host-down alert and the six "service unreachable" alerts trailing it as the same incident, and suppressed the trailing six. The signal survived and the noise stopped. That's PagerDuty automation working exactly as intended.

The half that rule couldn't touch is the subject of the honest section at the end. First, the work. This is a hands-on guide to tuning PagerDuty with only its native features: Event Orchestration, alert grouping, service dependencies, Incident Workflows, status updates, Webhooks v3, and the analytics that tell you whether any of it moved MTTR. No add-ons, no agents.

In part one of this series we mapped where on-call toil concentrates even when PagerDuty runs well. This part is how far the built-in tools take you against it, feature by feature, with configs you can paste. Part three covers the layer that sits on top.

Build in this order. Mobilizing people around a noisy incident stream only wastes more people, so the filtering comes first.

01Filter

Orchestration

Route, suppress and dedupe events before an incident exists.

02Fold

Grouping

Related alerts collapse into one incident with one owner.

03Relate

Dependencies

The service graph surfaces concurrent incidents next door.

04Mobilize

Workflows and updates

Responders, stakeholders and status in one step.

05Extend

Webhooks v3

Signed lifecycle events into your own automation.

The native stack in the order you should build it. Fewer, better incidents first; mobilization and glue after.

Event Orchestration: routing, deduplication and suppression

Event Orchestration is where most of your page-reduction wins live. It sits in front of a service (or globally, across an account) and runs events through an ordered set of rules before an incident is ever created (PagerDuty docs). Find it under Automation → Event Orchestration.

Rules are if condition then actions. Conditions use PagerDuty Condition Language (PCL) against the incoming event payload. A few you'll write constantly:

event.severity matches 'critical'
event.custom_details.namespace matches 'production'
event.source matches part 'db-'
event.summary matches regex 'disk.*(9[0-9]|100)%'

Routing. Send events to the right service instead of one catch-all. A rule on your global orchestration:

IF   event.custom_details.team matches 'payments'
THEN route to service "Payments On-Call"

Suppression: the afternoon win. This is the rule that halves pages. When a dependency fails, you get one real cause and a cascade of symptoms. Suppress the symptoms:

IF   event.summary matches part 'unreachable'
AND  event.custom_details.cluster matches 'prod-east'
THEN suppress (alert is recorded on the service, no incident triggers)

A suppressed alert is still recorded on the service, so you can see it, but it doesn't page anyone. Pair suppression with alert grouping (next section) so the surviving cause-alert absorbs the rest.

Time-based nesting. Orchestration rules nest, and PCL has a time condition, so you can gate an action on business hours:

IF   event.severity matches 'warning'
THEN (nested) IF now in Mon,Tue,Wed,Thu,Fri 09:00:00 to 18:00:00 Europe/London
              THEN route to service "Daytime Triage"
              ELSE suppress

Test every rule before you trust it. Paste a sample event into the orchestration editor's test view and confirm which rule matches and what it does. Don't skip this: a too-broad matches part on error will happily suppress the one page that mattered.

Alert grouping: collapse the cascade into one incident

Where suppression removes noise, alert grouping keeps the alerts but folds them into one incident, so one responder owns the whole cascade. On a service, open Settings → Alert Grouping and choose Intelligent (PagerDuty groups related alerts using learned patterns) or Content-Based for deterministic control (PagerDuty docs).

Content-based grouping is the one to reach for when you know the shared field:

Group alerts if the following fields match: custom_details.incident_key
Time window: 30 minutes (rolling)

That turns "database is down" plus fourteen downstream timeouts into one incident with fifteen alerts attached, not fifteen pages. Part one's section on related pages arriving as separate incidents is exactly the toil this removes.

Service dependencies and related incidents

PagerDuty can model which services depend on which and use that graph to surface related incidents, so the person paged for "checkout failing" sees that payments-db is also on fire. Configure it from a service's dependencies view in the service directory, or through the API (PagerDuty docs):

curl -X POST 'https://api.pagerduty.com/service_dependencies/associate' \
  -H 'Authorization: Token token=YOUR_API_TOKEN' \
  -H 'Accept: application/vnd.pagerduty+json;version=2' \
  -H 'Content-Type: application/json' \
  -d '{
    "relationships": [{
      "supporting_service": { "id": "PXXXXXX", "type": "service" },
      "dependent_service":  { "id": "PYYYYYY", "type": "service" }
    }]
  }'

Use business_service as the type when one side is a business service. Once the graph exists, an open incident's Related Incidents panel shows concurrent incidents on adjacent services, ranked by relatedness. It doesn't tell the responder which one is the root cause. It narrows the field of suspects. That distinction matters, and we'll come back to it.

Incident Workflows: one step to mobilize

An Incident Workflow bundles the manual steps a responder repeats on every major incident: add responders, notify a stakeholder list, start a conference bridge, set priority. Build them under Automation → Incident Workflows (PagerDuty docs). Workflows replaced the older Response Plays, which reached end of life in 2024; if your account still has plays, enabling Incident Workflows converts them (migration guide).

A typical Sev-1 workflow:

Add the database and platform on-call as responders
Subscribe the #incident-stakeholders channel and the leadership status list
Set priority to P1
Post the conference bridge link into the incident

You can start a workflow by hand from the incident, or give it a conditional trigger (for example, priority is P1 on these services) so it runs on its own. That automatic trigger is powerful: a P1 on a critical service can mobilize the whole response before the primary responder acknowledges.

Status updates and stakeholder comms

Half of incident toil is telling people what's happening. PagerDuty splits that into two audiences. Responders work the incident. Stakeholders (subscribers) only need status. Enable Status Updates on a service or business service and let stakeholders subscribe (PagerDuty docs).

Post updates through the API so they go out consistently:

curl -X POST 'https://api.pagerduty.com/incidents/PINCIDENT/status_updates' \
  -H 'Authorization: Token token=YOUR_API_TOKEN' \
  -H 'Accept: application/vnd.pagerduty+json;version=2' \
  -H 'Content-Type: application/json' \
  -H 'From: you@example.com' \
  -d '{ "message": "Identified: failed DB failover in prod-east. Mitigating now. Next update in 20m." }'

A status update posted early also answers part one's escalation problem: a responder who says "investigating, next update in 15 minutes" is visibly working, and nobody escalates "just in case." For customer-facing signals, wire a business service to a Status Page so subscribers get updates without anyone hand-writing an email per incident.

Webhooks v3: custom automation

When native actions run out, Webhooks v3 push incident lifecycle events to your own endpoint so you can automate anything: open a Jira ticket on incident.triggered, kick a runbook on incident.acknowledged, archive a timeline on incident.resolved (PagerDuty docs).

Create a subscription scoped to one service:

curl -X POST 'https://api.pagerduty.com/webhook_subscriptions' \
  -H 'Authorization: Token token=YOUR_API_TOKEN' \
  -H 'Accept: application/vnd.pagerduty+json;version=2' \
  -H 'Content-Type: application/json' \
  -d '{
    "webhook_subscription": {
      "type": "webhook_subscription",
      "delivery_method": {
        "type": "http_delivery_method",
        "url": "https://hooks.example.com/pagerduty"
      },
      "events": [
        "incident.triggered",
        "incident.acknowledged",
        "incident.resolved"
      ],
      "filter": { "type": "service_reference", "id": "PXXXXXX" }
    }
  }'

Verify every delivery. V3 signs each payload with an X-PagerDuty-Signature header (HMAC-SHA256 over the raw body with your subscription secret). Reject anything that doesn't match: an unverified webhook endpoint is an open door for fake incidents. And remember the receiver is now production code you own, on-call included.

Reading MTTA and MTTR from Analytics

None of the above is worth anything if you can't tell whether it worked. Analytics → Incidents reports mean time to acknowledge (MTTA) and mean time to resolve (MTTR) per service, team and time window (PagerDuty docs). Pull the aggregate through the API to trend it:

curl -X POST 'https://api.pagerduty.com/analytics/metrics/incidents/all' \
  -H 'Authorization: Token token=YOUR_API_TOKEN' \
  -H 'Accept: application/vnd.pagerduty+json;version=2' \
  -H 'Content-Type: application/json' \
  -d '{
    "filters": {
      "created_at_start": "2026-06-01T00:00:00Z",
      "created_at_end":   "2026-07-01T00:00:00Z"
    },
    "aggregate_unit": "month"
  }'

How to read it. Watch three numbers after a tuning pass. Incident count should drop where you added suppression and grouping. That's your noise win, and it translates directly to fewer 3 a.m. pages. MTTA usually improves when routing gets people to the right service faster. MTTR is the stubborn one. Plays and status updates shave coordination overhead, but the biggest chunk of MTTR is investigation time, and nothing in this guide touches it. Keep pages per on-call per week (part one's load metric) on the same view: falling incident count with flat MTTR tells you exactly where the remaining work is.

One incident through the native stack

Here's what a well-tuned setup looks like at 02:14. A checkout-api high-error-rate alert arrives. Orchestration routes it to Payments On-Call and suppresses three "unreachable" symptoms. Content-based grouping folds four more alerts into the same incident. Related Incidents shows that payments-db also has an open incident. The Sev-1 play adds the database on-call and opens a bridge. Everything native did its job, and it was a lot.

Now the responder has to work out what happened. There are three reasonable readings, and each predicts something different.

01Reading 1

Its own failure

checkout-api broke on its own: errors start there, dependencies stay healthy.

02Reading 2

Database failover

payments-db is also open; the failover explains every downstream timeout.

03Reading 3

A change upstream

A deploy or config change shortly before the first alert broke a shared dependency.

Three readings of the same grouped incident. The Related Incidents panel supports reading two; it can't confirm or rule out any of them.

None of them is answered inside PagerDuty. Its own failure needs checkout-api's error rate and logs from your monitoring tool, checked against the healthy state of its dependencies. A database failover needs the database's replication and failover events from the cloud provider or the database itself, and a check that checkout-api's errors start after the failover, not before. A change upstream needs the deploy history and config changes for every service in the dependency graph over the previous hour or two.

On this night it was reading three: a config change to the database's connection settings two hours earlier, which made the failover job fail at 02:14. The Related Incidents panel pointed at the right neighbor. The cause was two hours back, in a tool PagerDuty never sees.

What each native tool solves, and what it leaves

Native tool	What it automates	What it still leaves to a person
Event Orchestration	Routing, suppression, time-based rules	Writing rules broad enough, but not too broad
Alert grouping	One incident per cascade	Knowing which alert in the group is the cause
Service dependencies	Related incidents next door	Deciding which related incident started it
Incident Workflows	Mobilizing responders and stakeholders	Investigating once everyone is on the bridge
Status updates	Consistent stakeholder comms	Knowing what to say about the cause
Webhooks v3	Delivery to your own code	Writing, securing and on-calling that code

The right-hand column has one pattern. Native PagerDuty automation acts on the alert stream and the people. Anything that needs reading infrastructure evidence and choosing the next check stays with the on-call.

The ceiling: orchestration decides who gets paged, not what's wrong

Everything above is worth doing. It's also worth being precise about what it does.

01Orchestration

Who and when

Decides who gets paged and how often. Not what is wrong.

02Related incidents

Narrows suspects

Shows what else is on fire. Not which fire started it.

03Workflows

Mobilize people

Everyone joins the bridge, then waits for someone to investigate.

04Webhooks

Your code

Anything is possible, and you write, secure and on-call all of it.

What native PagerDuty automation can't do, however well you build it.

Orchestration works on alerts. Suppression, grouping, routing and dependencies make the right person's phone buzz once, with a deduplicated signal. That's valuable, and it's the whole game for these tools.

Related incidents narrow suspects. They don't diagnose. The dependency graph says "payments-db is also down." It can't say the failover job died at 02:14 because a config change two hours earlier flipped a setting. Someone still has to go look.

Plays mobilize. They don't investigate. A Sev-1 play assembles the whole response and a bridge in one step. Then everyone on the bridge waits for the same thing: a person to open a laptop, pull the metrics around the alert window, check recent deploys and form a hypothesis.

That's the structural ceiling. PagerDuty is excellent at getting the right person paged once, with context about which services are involved. It does nothing between the page and that person's first query. The clock that dominates MTTR, from acknowledgment to root cause, starts when someone wakes up, and no native rule shortens it.

The approval question you'll meet again

The webhook receiver is where most teams first wire PagerDuty to something that acts: restart a worker on incident.triggered, roll back on a specific alert. That forces a decision you'll see again in part three: which actions run unattended and which wait for a person. In part three, CloudThinker acts within Auto Mode, so you make the same call: what runs on its own and what waits for approval. If you've already decided which restarts your webhook may run unattended, you've done half the work.

What comes next

The gap native tooling can't close is the investigation itself, the minutes between the page and the diagnosis. That's where a layer on top of PagerDuty earns its place: CloudThinker picks up a triggered incident, investigates the underlying infrastructure straight away, and posts a diagnosis with evidence into the incident notes while the person is still waking up. Your escalation policies stay untouched and people stay paged. That's part three of this series.

To try it against your own incidents first, ask something like this in your first session:

"Last night's checkout-api incident had a related incident on payments-db. Check deploys and config changes on both services and their dependencies for the two hours before 02:14, and tell me which one started it. Propose only, change nothing."

Try CloudThinker free with 100 premium credits and no card required, or continue with part three.