Building an Incident Response Workflow with Runbooks and Native Tools

It's 3:12 a.m. and APIHighP99Latency has paged you. You open the incident on your phone. It says the alert name, a value of 2.3s and nothing else. You get to a laptop, open Grafana, find the right dashboard on the third try, check whether anyone deployed, search Slack for the last time this fired, and twenty minutes in you're still not sure whether customers are affected.

In Part 1 we broke down why root cause analysis takes hours: scattered context, tribal knowledge, and the "what changed?" question nobody can answer at 3 a.m. This article is the DIY fix. You'll build an incident response runbook system with tools you almost certainly already run: Prometheus Alertmanager, PagerDuty or Grafana OnCall, and your CI pipeline. No new vendors, no procurement cycle.

The goal is narrow and concrete. When an alert fires, the responder should land on a page that tells them what to check, in what order, and when to escalate, instead of starting from a blank Slack channel. If you want the wider context on RCA methods and where runbooks fit, the root cause analysis guide covers the full landscape. Here, we build.

---

Why ad hoc incident response breaks

Most teams already have runbooks. They still lose the first twenty minutes of every incident, for structural reasons:

Runbooks are written as documentation, not procedures. A wiki page titled "API latency" that explains how the service works isn't a runbook. A stressed responder needs an if-this-then-that procedure they can run without understanding the whole system.

The link isn't on the page. If the responder has to search a wiki for the right document, the runbook effectively doesn't exist at 3 a.m. Delivery matters as much as content.

"What changed?" has no single answer. Deploys live in CI, flag flips in the flag service, infra applies in Terraform Cloud. Reconstructing the change timeline by hand is where most RCA time goes.

Escalation lives in someone's head. Without a time budget written down, the responder keeps going solo until it's obviously too late.

Each of these has a fix you can ship this quarter with the tools you already have.

---

What a runbook has to contain

01Trigger

One alert

The exact alert name and threshold. One runbook per alert, not per service.

02Severity and impact

Two minutes

How to tell a page from noise, and the first check for real customer impact.

03Decision tree

Ordered checks

Most likely and fastest cause first, each with an explicit yes or no branch.

04Verify and escalate

A number

A query that proves recovery, a time budget, and who to page next.

05Rollback

The exact command

Plus the preconditions, or a clear statement that rollback is unsafe.

06Last verified

A date and a name

The only defense against instructions that are confidently wrong.

Every entry in your incident response process needs seven parts:

1. Trigger. The exact alert name and the threshold that fired it. One runbook per alert, not one runbook per service. If two alerts share a runbook, the runbook is too vague.
2. Severity guidance. How to tell a page-worthy incident from noise, in terms the responder can evaluate in two minutes. "P1 if error rate is also elevated, P3 if latency is high but errors are flat."
3. Impact check. The first query or dashboard that answers "are customers actually affected?" This determines everything downstream: comms, escalation, urgency.
4. Decision tree. Ordered checks, each with an explicit "if yes, do X; if no, continue" branch. Order them by probability times speed: the most common cause that's fastest to check goes first. For most latency incidents, that check is "did we just deploy?"
5. Verification steps. How to confirm the mitigation worked. "Latency recovered" means a specific query returning a specific range, not a vibe.
6. Escalation. Who to page next, at what point, with what context. A time budget ("if not mitigated in 20 minutes, escalate") prevents the solo-hero-at-4-a.m. failure mode.
7. Rollback. The exact command to undo the most recent change, plus any preconditions. If rollback isn't safe for this service (migrations, queue consumers), the runbook must say so explicitly. This is the single most expensive thing to discover mid-incident.

Two writing rules matter more than any template. First, every command must be copy-pasteable: real namespaces, real label selectors, placeholders clearly marked. A responder at 3 a.m. will paste exactly what you wrote. Second, write for the newest engineer on the rotation, not the person who built the service. If a step assumes knowledge, link it or inline it.

---

A runbook template you can copy today

Here's a complete SRE runbook for a realistic alert. Replace the placeholders, drop it in your repo (runbooks belong in git, next to the alert definitions, more on why below), and use it as your runbook template for every other alert.

# Runbook: APIHighP99Latency

**Alert:** `APIHighP99Latency`: API p99 latency above 1.5s for 10 minutes
**Service:** api-gateway (owns: platform team)
**Dashboard:** https://grafana.example.com/d/api-overview
**Last verified:** 2026-06-30 by @lena

## Severity

- **P1**: p99 above 1.5s AND 5xx rate above 1%. Customers are timing out. Page secondary immediately.
- **P2**: p99 above 1.5s, error rate flat. Degraded but functional. Work it solo, 20-minute budget.
- **P3**: fired during a known batch window (02:00 to 02:30 UTC). Ack, verify it recovers, file a ticket to tune the alert.

## Impact check (2 minutes)

Is the latency reaching users as errors? Check the 5xx ratio over the last 5 minutes:

```bash
curl -sG 'https://prometheus.example.com/api/v1/query' \
  --data-urlencode 'query=sum(rate(http_requests_total{job="api-gateway",code=~"5.."}[5m])) / sum(rate(http_requests_total{job="api-gateway"}[5m]))'
```

Above 0.01 (1%) means customers are failing: treat as P1 and post in #incidents before continuing.
Also glance at the dashboard panel "Requests by status" to see when the climb started.

## Decision tree

### 1. Did we just deploy? (most common cause)

Check the deploy annotations on the dashboard, or:

```bash
kubectl -n api rollout history deployment/api-gateway | tail -5
```

- **Deploy in the last 60 minutes: go to Rollback.** Don't debug first. Roll back, then debug.
- No recent deploy: continue.

### 2. Is one pod the outlier?

```bash
kubectl -n api top pods -l app=api-gateway --sort-by=cpu
```

- **One pod hot while others are idle:** delete it and let the ReplicaSet reschedule:

  ```bash
  kubectl -n api delete pod <pod-name>
  ```

- All pods equally loaded: continue.

### 3. Is the database the bottleneck?

Check panel "DB connection pool saturation" on the dashboard.

- **Pool above 90%:** this is the downstream `postgres-primary` runbook's territory. Follow `runbooks/postgres-connection-saturation.md`.
- Pool healthy: continue.

### 4. Is traffic anomalous?

Check panel "Requests by client". A single API key above 3x its baseline means a customer script or an abuse pattern.

- **Yes:** apply the rate-limit override per `runbooks/rate-limit-override.md`.
- No: you're past the common causes. Escalate now rather than at the time budget.

## Rollback

Safe for this service (stateless, no migrations in the deploy pipeline):

```bash
kubectl -n api rollout undo deployment/api-gateway
kubectl -n api rollout status deployment/api-gateway --timeout=180s
```

## Verification

Run after any mitigation. Recovery means p99 back under 400ms for 10 consecutive minutes:

```bash
curl -sG 'https://prometheus.example.com/api/v1/query' \
  --data-urlencode 'query=histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket{job="api-gateway"}[5m])) by (le))'
```

Then confirm the alert resolves in Alertmanager. Don't manually silence it to make it green.

## Escalation

- 20 minutes without mitigation, or any P1: page `platform-secondary` via the escalation policy.
- Database suspected: page `dba-oncall` with the pool saturation graph attached.
- Say what you ruled out as well as what is broken.

Notice what this template optimizes for. The impact check is a query with a threshold, so severity is decided by data, not by how the graph feels. The deploy check is step one of the decision tree, rollback is a first-class section, and verification is a query with a number, not "confirm things look better."

The same runbook at 3:12 a.m.

01Page

Alert fires

The phone notification carries the runbook link.

02Impact

Is it real?

A 5xx rate query decides P1, P2 or P3 in two minutes.

03Change

Did we deploy?

The deploy marker on the graph answers it at a glance.

04Mitigate

Roll back

Roll back first, debug second, when the deploy lines up.

05Verify

p99 under 400ms

Ten consecutive minutes, then the alert resolves on its own.

Walk the opening page through it. The phone notification has a Runbook link (wiring below). The impact query returns 0.004: 0.4% 5xx, so this is P2, degraded but functional, 20-minute budget, no need to wake anyone else. Step one: the latency graph shows a deploy annotation at 03:04, eight minutes before the page. That lines up, so the runbook says roll back before debugging. rollout undo completes in 90 seconds, the verification query drops back to 310ms and stays there, and the alert resolves by itself at 03:26. The responder files a ticket against the 03:04 commit and goes back to bed.

The point isn't that this incident was hard. It's that every decision in it was written down in advance, so the responder spent 14 minutes executing instead of 40 minutes reconstructing.

---

Wire the alert to the runbook

01Detect

Prometheus

• Alerting rulerunbook_url annotation on every page
• promtool in CIFails the build without one

The link travels with the alert

02Route and page

Alertmanager

• PagerDutylinks block, Event Orchestration notes
• Grafana OnCallRoutes and escalation chains in Terraform
• SlackOpen runbook button

Runbook one tap from the page

03Respond

Git and Grafana

• Runbooks in gitNext to the alert definitions
• Deploy annotationsA vertical line at every change

What changed, at a glance

A runbook nobody can find is a runbook that doesn't exist. The link must be on the page itself, not in a wiki the responder has to search. Every major alerting stack supports this natively.

Prometheus Alertmanager: the runbook_url annotation

The convention, used by kube-prometheus and most public mixins, is a runbook_url annotation on the alerting rule. Put it in the rule, and it travels with the alert everywhere Alertmanager sends it:

groups:
  - name: api-gateway
    rules:
      - alert: APIHighP99Latency
        expr: |
          histogram_quantile(0.99,
            sum(rate(http_request_duration_seconds_bucket{job="api-gateway"}[5m])) by (le)
          ) > 1.5
        for: 10m
        labels:
          severity: page
          team: platform
        annotations:
          summary: 'API p99 latency above 1.5s for 10 minutes'
          runbook_url: 'https://github.com/acme/runbooks/blob/main/api/high-p99-latency.md'

Then make Alertmanager surface it. The route and receiver structure below follows the official configuration reference. The links block on pagerduty_configs (Alertmanager 0.25 or later) turns the annotation into a clickable link on the PagerDuty incident. The Slack receiver assumes slack_api_url is set in the global block:

route:
  receiver: slack-default
  group_by: ['alertname', 'team']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  routes:
    - matchers:
        - severity="page"
      receiver: pagerduty-platform

receivers:
  - name: pagerduty-platform
    pagerduty_configs:
      - routing_key_file: /etc/alertmanager/secrets/pd-routing-key
        severity: 'critical'
        links:
          - href: '{{ (index .Alerts 0).Annotations.runbook_url }}'
            text: 'Runbook'
        details:
          runbook: '{{ .CommonAnnotations.runbook_url }}'
          firing: '{{ template "pagerduty.default.instances" .Alerts.Firing }}'

  - name: slack-default
    slack_configs:
      - channel: '#alerts'
        title: '{{ .CommonAnnotations.summary }}'
        text: 'Runbook: {{ .CommonAnnotations.runbook_url }}'
        actions:
          - type: button
            text: 'Open runbook'
            url: '{{ .CommonAnnotations.runbook_url }}'

Enforce the annotation in CI: a promtool check rules step plus a small script that fails the build if any rule with severity: page lacks runbook_url. That one gate is what keeps coverage from silently decaying as people add alerts. A minimal version with yq:

promtool check rules rules/*.yaml
missing=$(yq -r '.groups[].rules[]
  | select(.labels.severity == "page" and (.annotations.runbook_url // "") == "")
  | .alert' rules/*.yaml)
if [ -n "$missing" ]; then
  echo "Paging alerts without runbook_url:"; echo "$missing"; exit 1
fi

PagerDuty: Event Orchestration

If some alerts reach PagerDuty from sources you don't control as tightly as Alertmanager (Datadog monitors, CloudWatch, custom webhooks), use Event Orchestration to normalize them. An orchestration rule can match on event content and add a note carrying the runbook link, so the incident always shows it regardless of sender.

In the orchestration UI (Automation > Event Orchestration), create a rule on the service orchestration:

• Condition: the event summary contains p99 latency (check the condition builder for the exact operator your account offers)
• Action, Annotate: add a note with a templated value pulled from the event payload:

Runbook: {{event.custom_details.runbook}}

Because the Alertmanager config above copies runbook_url into details.runbook, the note resolves to the real URL. For senders that use the Events API v2 directly, skip orchestration and put the link in the payload. The API accepts a links array natively:

curl -s https://events.pagerduty.com/v2/enqueue \
  -H 'Content-Type: application/json' \
  -d '{
    "routing_key": "YOUR_INTEGRATION_KEY",
    "event_action": "trigger",
    "payload": {
      "summary": "API p99 latency above 1.5s",
      "source": "prometheus",
      "severity": "critical"
    },
    "links": [
      {
        "href": "https://github.com/acme/runbooks/blob/main/api/high-p99-latency.md",
        "text": "Runbook: APIHighP99Latency"
      }
    ]
  }'

Either way, the responder's phone notification opens onto an incident with the runbook one tap away.

Grafana OnCall: routes and escalation chains

If you're on the Grafana stack, Grafana OnCall handles the paging side. Alert annotations, including runbook_url, pass through the Alertmanager integration into the alert group details. What you configure in OnCall is the routing and escalation: which alerts page whom, and what happens when nobody acks.

Escalation chains are configurable in the UI, but they belong in Terraform for the same reason runbooks belong in git: review and history. The resource names below come from the Grafana Terraform provider; check attribute names against the provider version you pin.

resource "grafana_oncall_escalation_chain" "platform_critical" {
  name = "platform-critical"
}

resource "grafana_oncall_escalation" "notify_primary" {
  escalation_chain_id          = grafana_oncall_escalation_chain.platform_critical.id
  type                         = "notify_on_call_from_schedule"
  notify_on_call_from_schedule = grafana_oncall_schedule.platform_primary.id
  position                     = 0
}

resource "grafana_oncall_escalation" "wait_five_minutes" {
  escalation_chain_id = grafana_oncall_escalation_chain.platform_critical.id
  type                = "wait"
  duration            = 300
  position            = 1
}

resource "grafana_oncall_escalation" "notify_secondary" {
  escalation_chain_id          = grafana_oncall_escalation_chain.platform_critical.id
  type                         = "notify_on_call_from_schedule"
  notify_on_call_from_schedule = grafana_oncall_schedule.platform_secondary.id
  position                     = 2
}

resource "grafana_oncall_route" "critical_route" {
  integration_id      = grafana_oncall_integration.alertmanager.id
  escalation_chain_id = grafana_oncall_escalation_chain.platform_critical.id
  routing_regex       = "\"severity\":\\s*\"page\""
  position            = 0
}

That encodes the escalation section of your runbook (primary paged immediately, secondary after five unacked minutes) as configuration instead of prose someone has to remember to follow.

---

Make "what changed?" answerable in one glance

Step one of the decision tree was "did we just deploy?", because change is the leading cause of incidents, and Part 1 covered how much RCA time goes into reconstructing the change timeline. The fix is mechanical: every deploy posts a marker to the same place responders look at graphs.

Grafana's annotations API makes this a one-liner from any CI system. As the last step of your deploy job:

# GitHub Actions: final step of the deploy job
- name: Post deploy marker to Grafana
  if: success()
  env:
    GRAFANA_URL: https://grafana.example.com
    GRAFANA_TOKEN: ${{ secrets.GRAFANA_SA_TOKEN }}
  run: |
    curl -sf -X POST "$GRAFANA_URL/api/annotations" \
      -H "Authorization: Bearer $GRAFANA_TOKEN" \
      -H "Content-Type: application/json" \
      -d "{
        \"tags\": [\"deploy\", \"api-gateway\", \"production\"],
        \"text\": \"Deploy ${GITHUB_SHA:0:7} by ${GITHUB_ACTOR}: ${GITHUB_SERVER_URL}/${GITHUB_REPOSITORY}/commit/${GITHUB_SHA}\"
      }"

Omitting the time field timestamps the annotation at the moment of the call, which is what you want. Then enable the deploy tag as an annotation query on your service dashboards, and every latency graph gets a vertical line at each deploy. The 40-minute archaeology session from Part 1 becomes a glance: latency broke upward two minutes after the vertical line, done.

Do the same for anything else that changes production: feature flag flips, infrastructure applies, database migrations. Use a different tag per change type. The discipline generalizes: any system that can change production must announce that it did, in the place where symptoms are observed.

---

Wiki runbooks vs wired runbooks

The deeper difference is that the wired version turns incident response into a pipeline you can review and test. Runbooks get diffs and reviewers, missing links fail a build, and escalation is configuration. You still need a human to run it, which is the one part this setup can't change.

---

The honest limits

01Rot

Runbooks drift

Services get renamed and the safe rollback stops being safe.

02Execution

Humans at 3 a.m.

The slowest, most variable part of the system runs every step.

03Coverage

Only the known

The next hard incident is the one nobody wrote a runbook for.

This setup is worth building. It'll measurably shorten your incidents. It also has three failure modes that no amount of diligence fully removes, and you should go in knowing them.

Runbooks rot. The runbook describes the system as it was the day someone wrote it. Services get renamed, dashboards move, the "safe" rollback stops being safe after someone adds a migration step. The Last verified field in the template exists because unverified runbooks decay into a special kind of hazard: instructions that are confidently, specifically wrong. A quarterly review ritual helps. In practice, review loses to feature work, and you find the rot during the incident.

Humans execute them at 3 a.m. A runbook is a program whose runtime is a sleep-deprived person under pressure. They skip steps, paste commands into the wrong terminal and misread branch conditions, precisely because the situations where runbooks matter most are the ones where human execution is least reliable. Good formatting and copy-pasteable commands reduce the error rate. They don't remove the human from the loop, and the human is the slowest and most variable component in the system. This is the gap the runbook automation ladder exists to close: turning steps a human executes into steps a machine executes.

Coverage is never complete. You'll write runbooks for the incidents you've already had. The incident that pages you next quarter is, with grim reliability, the one without a runbook, because if it matched a known pattern you'd probably have alerted on the precondition and prevented it. Runbooks compress the known and stay silent on the novel. Expect them to handle the recurring 60 to 70% of pages and to leave your hardest incidents exactly as hard as before.

None of this is a reason to skip the work. A rotting runbook plus a deploy marker still beats a blank terminal. But be honest about what you've built: a paper procedure with good delivery, executed by hand.

---

What comes after paper

You now have a working incident response process: structured runbooks in git, links delivered on the alert, deploy markers answering "what changed." The obvious next question is the one the limits section raises. If the runbook is a decision procedure, and the procedure is written down, why is a human at 3 a.m. still the execution engine?

That's Part 3: what happens when an agent executes the investigation instead. It reads the same runbooks, checks the same deploy markers and hands you a findings summary instead of a checklist. Under Auto Mode it starts in Manual: reads run freely, and every write (the rollback, the pod delete, the rate-limit override) waits for a named person's approval. You move individual actions to Auto only when a runbook you wrote covers them, and anything outside that still comes to a human.

The runbooks from this post are exactly what the agent needs. In plain language, the first session looks like:

"Here's our runbook for APIHighP99Latency. Next time it fires, run the impact check and the decision tree, and tell me which branch you landed on and why. Propose the mitigation, change nothing."

"List every paging alert in our Prometheus rules that has no runbook_url, and draft a first runbook for the three that fired most last month."

If you want to see that in practice first, CloudThinker's free tier includes 100 premium credits, no card required.

Related reading

• Why root cause analysis takes hours (Part 1)
• Automated RCA with an AI SRE agent (Part 3)
• The root cause analysis guide
• The runbook automation ladder