Alert Fatigue: A Triage System That Lets On-Call Sleep

Alert fatigue is what happens when monitoring pages people about things that don't matter, often enough that they stop trusting the pager. The sequence is predictable. An engineer is woken at 3 a.m. for a disk at 81 percent on a node that self-heals. She acknowledges it, mutes the channel, and three weeks later sleeps through the page that mattered. Nobody lacked discipline. The system had no triage, and you can engineer most of that back in.

This article builds a three-layer alert triage system: severity-based routing, deduplication and correlation, and auto-triage. The examples use Prometheus and Alertmanager because that's the most common open-source stack, but the design carries over to Datadog monitors, CloudWatch alarms, or anything that can put labels on an alert. Triage is the front door to diagnosis. Our root cause analysis guide covers what happens after the page fires. This one is about making sure the page deserves to fire at all.

---

What alert fatigue looks like up close

Before fixing it, name the failure modes. Most rotations suffer from three.

Pages that aren't actionable. A page is a demand that someone get out of bed and do something. If the alert has no action attached ("CPU is high", "error rate elevated on a service nobody owns any more"), the responder acknowledges it and goes back to sleep, and every one of those teaches the team that pages can be ignored. Audit the last 30 days of pages and count how many ended with a human doing something a machine couldn't. Teams that have never run this audit commonly find that only 20 to 40 percent of pages were actionable.

Duplicate storms. One database failover fires the database-down alert, latency alerts on eight services that call it, error-budget alerts on four SLOs, and a synthetic check per region. That's 40 notifications for one cause. The on-call spends the first fifteen minutes acknowledging noise, and the one alert that names the cause is buried in the middle.

Severity inflation. Every team wants its service taken seriously, so everything becomes critical. When 90 percent of alerts are critical, the word routes nothing, and routing decisions get made by a tired human at 3 a.m. instead of by configuration. This is a governance problem. Alertmanager tuning won't fix it unless a rubric is enforced in code review.

01Not actionable

Ack and sleep

The page has no action attached, so it teaches the team that pages can be ignored.

02Duplicate storms

40 for 1

One failover pages for the database, eight callers, four SLOs and every region.

03Severity inflation

All critical

When nine alerts in ten are critical, the label stops routing anything.

Why the usual fixes don't hold

Most teams have tried something already. It tends to fail for structural reasons, not effort.

• Raising thresholds one at a time. It quiets the alert that annoyed someone this week and does nothing about the storm shape. Forty alerts with higher thresholds are still forty alerts when the database really fails.
• Muting channels. It cuts volume by cutting attention. The real page lands in the same muted channel.
• Per-alert rules with no shared labels. Grouping and inhibition match on labels. If half the rules call the node node and the other half instance, correlation silently does nothing.
• Tuning once. Thresholds decay as traffic, latency and memory baselines move. A clean alert set in January is noisy by June without anyone changing it.

The fix for all of this has the same shape: decide severity by rubric, route by severity, collapse duplicates by cause, and attach enough context to every alert that triage takes seconds.

01Layer 1

Route by severity

page, ticket or info, set in the rule and reviewed in the same PR.

02Layer 2

Collapse by cause

Grouping, inhibition and flap control turn a storm into one notification.

03Layer 3

Enrich each alert

Runbook, owner and clean auto-resolve, so triage takes seconds.

---

Layer 1: severity-based routing

A severity rubric that survives arguments

Three tiers are enough. Past three, people can't tell them apart.

The one rule that matters: every page must be actionable and urgent. Both. Actionable but not urgent is a ticket. Urgent-looking but not actionable ("load is high") is either info or a broken alert that should be rewritten against a symptom users feel. If it can wait until morning, it's a ticket. Enforce that one sentence and most overnight noise goes away for most teams.

Set severity as a label on the alerting rule itself, so it's reviewed in the same pull request as the threshold:

groups:
  - name: api-slo
    rules:
      - alert: APIHighErrorRate
        expr: |
          sum(rate(http_requests_total{job="api", code=~"5.."}[5m]))
            / sum(rate(http_requests_total{job="api"}[5m])) > 0.05
        for: 5m
        labels:
          severity: page
          team: platform
        annotations:
          summary: 'API 5xx ratio above 5% for 5 minutes'
          runbook_url: 'https://runbooks.internal.example.com/api/high-error-rate'

Routing it in Alertmanager

The route tree turns the rubric into behavior. Timing differs per severity on purpose: pages notify fast and re-notify while unresolved, tickets batch up and repeat rarely.

route:
  receiver: ticket-queue
  group_by: ['alertname', 'cluster', 'service']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 12h
  routes:
    - matchers:
        - severity="page"
      receiver: pagerduty-oncall
      group_wait: 30s
      group_interval: 5m
      repeat_interval: 4h
    - matchers:
        - severity="ticket"
      receiver: ticket-queue
      group_wait: 10m
      group_interval: 30m
      repeat_interval: 24h
    - matchers:
        - severity="info"
      receiver: blackhole

receivers:
  - name: pagerduty-oncall
    pagerduty_configs:
      - routing_key_file: /etc/alertmanager/secrets/pagerduty-key
  - name: ticket-queue
    slack_configs:
      - api_url_file: /etc/alertmanager/secrets/slack-webhook
        channel: '#alerts-triage'
        send_resolved: true
  - name: blackhole

Why each choice:

• group_wait: 30s on the page route holds the first notification for 30 seconds to batch alerts arriving together. That's long enough to collapse a burst into one page and short enough not to slow the response.
• repeat_interval: 4h re-fires an unresolved page every four hours. Tickets repeat every 24 hours, because re-notifying hourly about something non-urgent is how channels get muted.
• The blackhole receiver has no integrations, which is valid Alertmanager config. Alerts routed there aren't sent anywhere but stay queryable in the Alertmanager UI and API. Info alerts exist for correlation and debugging, not notification.
• The fallback receiver is the ticket queue, not the pager. An unrouted alert is one nobody made a severity decision about, and undecided must never mean "wake someone."

You can check a routing tree before it ships. amtool shows which receiver a given label set lands on:

amtool config routes test --config.file=alertmanager.yml severity=page team=platform
# pagerduty-oncall
amtool config routes test --config.file=alertmanager.yml team=platform
# ticket-queue

Put that in CI next to amtool check-config and routing mistakes get caught in review instead of at 3 a.m.

---

Layer 2: deduplication and correlation

Routing decides who hears about an alert. This layer decides how many alerts they hear about.

Grouping

group_by above already does a first pass: alerts sharing alertname, cluster and service collapse into one notification. Tune the key to your topology. Grouping on ['alertname'] alone is too aggressive, because a latency alert on two unrelated services becomes one confusing notification. Grouping on every label including instance is too loose, because a 50-node rollout failure becomes 50 pages. The right key groups the alerts a single responder would treat as one problem.

Inhibition: suppress the downstream

Inhibition rules encode causality. When the cause-level alert fires, symptom-level alerts that share identifying labels are suppressed. The classic case is a dead node: once you know it's down, every alert about processes on it is redundant.

inhibit_rules:
  - source_matchers:
      - alertname="NodeDown"
    target_matchers:
      - severity=~"page|ticket"
    equal: ['cluster', 'node']
  - source_matchers:
      - alertname="DatacenterNetworkPartition"
    target_matchers:
      - severity=~"page|ticket"
    equal: ['cluster']
  - source_matchers:
      - severity="page"
    target_matchers:
      - severity="ticket"
    equal: ['alertname', 'cluster', 'service']

The first rule says: while NodeDown fires for a given cluster and node, suppress every page or ticket with the same cluster and node labels. The equal list keeps inhibition scoped. Without it, one dead node in staging would silence production. The third rule handles conditions with two thresholds: if a page-level and a ticket-level rule cover the same condition, the page inhibits the ticket so you're notified once.

Inhibition only works if your alerts carry consistent labels. If half your rules label the node node and half label it instance, the equal match silently fails. Label hygiene is boring, and it's the whole foundation of this layer.

Flap suppression

A metric oscillating around a threshold produces a fire, resolve, fire cycle that's worse than a steady alert, because each cycle is a fresh notification. Two Prometheus controls fix most of it:

• for requires the condition to hold continuously before the alert fires. for: 5m means a 30-second CPU spike never becomes an alert. Almost every rule should have one. The exceptions (hard down, data loss) are rare.
• keep_firing_for (Prometheus 2.42 and later) is the mirror image. The alert keeps firing for that long after the condition clears, so a metric bouncing across the line produces one continuous alert instead of a resolve and refire storm.

- alert: QueueDepthHigh
  expr: max_over_time(queue_depth{job="worker"}[5m]) > 10000
  for: 10m
  keep_firing_for: 5m
  labels:
    severity: ticket
    team: pipelines

Using max_over_time or avg_over_time in the expression is a third smoothing lever when the raw series is spiky by nature.

---

One failover, layer by layer

Here's the storm from the top of the article, worked through. At 03:12 the primary of the orders PostgreSQL cluster fails over. Within two minutes, 40 alerts arrive.

The readings. A storm like this can mean three different things, and each predicts something you can check:

• One upstream cause. Every alert started inside one propagation window (typically 1 to 5 minutes), the services involved actually call the database, and the database alert fired first. Expect the downstream alerts to resolve together when the database recovers.
• Two unrelated problems at once. Some alerts started well before the failover, or sit on services with no path to orders. Expect them to stay firing after the database recovers.
• A shared dependency below the database. A network partition or a dead node took the database and other things down together. Expect node or network alerts with an earlier start time.

The check. Start times and topology settle it. In Prometheus, ALERTS and ALERTS_FOR_STATE show what's firing and since when:

# Which alerts are firing right now, by severity and service
count by (alertname, service, severity) (ALERTS{alertstate="firing"})

# When each one first went pending (unix seconds)
ALERTS_FOR_STATE

Here, the PostgresPrimaryDown alert has the earliest timestamp, every latency and SLO alert started within 90 seconds after it, and all eight services list orders as a dependency. No node or network alerts fired. That's reading one.

The rule that encodes it. Grouping alone gets the 40 alerts down to roughly a dozen groups. Inhibition does the rest, but only if the downstream alerts carry a label that names the database they depend on. Add db_cluster to those rules, then:

inhibit_rules:
  - source_matchers:
      - alertname="PostgresPrimaryDown"
    target_matchers:
      - severity=~"page|ticket"
      - alertname!="PostgresPrimaryDown"
    equal: ['db_cluster']

01Raw

40 alerts

Database down, eight latency alerts, four SLO burns, synthetics per region.

02Grouped

~12 groups

group_by on alertname, cluster and service merges same-cause repeats.

03Inhibited

2 notifications

The database alert suppresses downstream pages that share its labels.

04Delivered

1 page

Titled by the upstream cause, with the rest attached as evidence.

Next time, the responder gets one page titled by the cause. The downstream alerts still exist in Alertmanager as evidence, but they don't notify. If one of them stays firing after the database is back, it wasn't part of this incident, and it'll page on its own once the source alert clears.

---

Layer 3: auto-triage patterns

The first two layers reduce volume. This layer reduces the cost of each alert that's left, which is where acknowledge-to-diagnosis time lives. We break that stage down in how to reduce MTTR.

Every alert carries its runbook. A runbook_url annotation on every rule means the responder clicks one link and lands on the diagnosis steps instead of searching a wiki at 3 a.m. Enforce it mechanically. A CI check that fails any alerting rule without runbook_url takes an afternoon to build and pays for itself the first weekend:

# Fails if any alerting rule lacks severity, team or runbook_url
yq -e '[.groups[].rules[] | select(.alert) |
  select((.labels.severity == null) or (.labels.team == null) or (.annotations.runbook_url == null))
  | .alert] | length == 0' rules/*.yml

If you don't have runbooks to point at yet, start with DIY incident response runbooks. A runbook set that covers only the top five alerts by frequency already covers most pages.

Every alert carries an owner. A team label does two jobs. It routes tickets to the right queue through child routes that match on team, and it makes orphaned alerts visible. An alert whose team maps to a squad that was reorganized away eighteen months ago is a deletion candidate, not a routing problem.

Auto-resolve, and treat alerts that don't resolve as bugs. Alertmanager marks an alert resolved when Prometheus stops sending it, and send_resolved: true closes the loop at the destination. If an alert routinely stays firing after the issue is fixed, the expression is wrong. It's usually an absent-metric problem or a threshold on a counter that never resets. Alerts that need manual cleanup turn into ambient noise on every dashboard.

Correlation into incidents. Grouping and inhibition work on label matching inside one Alertmanager. Real correlation goes further: recognizing that 40 alerts across Prometheus, CloudWatch and your synthetics provider are one incident with one cause. The heuristic is the one from the walkthrough: the alerts started within one propagation window, they share a dependency path, and one sits upstream of the rest. When those hold, the responder should see one incident titled by the upstream cause, with the downstream alerts attached as evidence. Inside one stack you can approximate this with disciplined group_by and inhibition. Across several monitoring tools it takes either an event-correlation layer or a human doing it in their head at 3 a.m., which is exactly the load this system exists to remove.

---

Why this beats the untriaged setup

The deeper difference is where the reasoning lives. In an untriaged setup, every alert asks the responder to work out severity, ownership and cause from scratch. A triage system moves the first two into configuration you review once. Cause is the part configuration can only approximate, because it depends on what else is happening right now.

---

The honest part: tuning is never done

Every threshold you set decays. Traffic grows, so the request-rate alert that meant "something is wrong" now fires every Tuesday peak. A service gets faster, so a latency threshold that was tight is now so loose it won't fire before users notice. Deploys move baseline memory. Nobody did anything wrong. The system underneath moved.

Treat tuning as scheduled work. A monthly 30-minute review of three lists keeps the decay in check:

1. The top ten alerts by firing count: candidates for re-thresholding or deletion.
2. Every page acknowledged without action: candidates for demotion to ticket.
3. Every incident with no preceding page: candidates for a new alert.

The first list is one query:

topk(10, sum by (alertname) (changes(ALERTS_FOR_STATE[30d])))

Teams that run this loop typically cut page volume by half within a quarter or two. Teams that don't drift back to noise in about the same time.

Keep the bigger point in view, though. Noise reduction treats symptoms. If the same alert fires every week, the durable fix is the cause of the recurrence, not a smarter threshold. A triage system buys you the calm to do that engineering. On-call burnout rarely comes from the worst incident of the quarter. It comes from the two hundred small interruptions between incidents, and the only permanent fix for those is making the failures stop.

Rules of thumb

• Every page must be actionable and urgent. Fail either test and it's a ticket or info.
• If it can wait until morning, it's a ticket. No "but it might get worse": write a page-severity rule for the bad condition instead.
• Never route unclassified alerts to the pager. Undecided severity defaults to the ticket queue.
• No alerting rule merges without severity, team and runbook_url. Enforce it in CI.
• Acknowledged without action is a signal. Twice in a row for the same alert means demote it or delete it.
• One cause should produce one notification. If a single failure pages you three times, you owe yourself a grouping key or an inhibition rule.
• Deleting an alert is a valid tuning outcome. An alert nobody has acted on in six months isn't a safety net. It's noise with a YAML file.

---

Where this stops scaling by hand

Everything above is buildable with Alertmanager and discipline. Two parts stay hard: correlation across tools (recognizing that CloudWatch, Prometheus and your APM are describing one incident) and keeping severity honest as systems change.

That's what CloudThinker Resolve takes on. Its anomaly detection catches drift before static thresholds fire, and event correlation collapses cross-tool storms into a single incident with the probable cause ranked on top, along with the start times and dependency path that support it. It reads your existing Alertmanager, PagerDuty and cloud monitoring, so the three layers you built stay in place.

The agent reads alerts, history and topology freely and proposes changes (a new inhibition rule, a demotion from page to ticket, a for clause on a flapping rule). Changes go through Auto Mode, so you decide what runs on its own and what waits for approval.

A few things to ask once your stack is connected:

"Which of our alerts paged in the last 30 days and were acknowledged with no action taken? Group them by team and recommend page, ticket or delete for each."

"Last night's orders database failover sent 40 alerts. Show me which ones started after the primary went down and share a dependency on it, and draft the inhibition rule that would have collapsed them. Propose only, change nothing."

"List alerting rules missing a severity, team or runbook URL, and who last changed each one."

You can try it on the free tier: 100 premium credits, no card required.

Related reading

• Root cause analysis guide
• How to reduce MTTR
• DIY incident response runbooks
• Automated root cause analysis with an AI SRE agent
• Auto Mode