Alert Fatigue: A Triage System That Lets On-Call Sleep

Alert fatigue is what happens when your monitoring pages people about things that do not matter, often enough that they stop trusting the pager. The failure mode is predictable: an engineer gets woken at 3 a.m. for a disk at 81 percent on a node that self-heals, acknowledges it, mutes the channel, and three weeks later sleeps through the page that actually mattered. This is not a discipline problem. It is a triage problem, and you can engineer your way out of most of it.

This article builds a three-layer alert triage system: severity-based routing, deduplication and correlation, and auto-triage patterns. The examples use Prometheus Alertmanager because it is the most common open-source stack, but the design applies to Datadog monitors, CloudWatch alarms, or anything else that can attach labels to an alert. Triage is also the front door to diagnosis — if you have not read our root cause analysis guide, it covers what happens after the page fires; this article is about making sure the page deserves to fire at all.

What alert fatigue actually looks like

Before fixing it, name the specific failure modes. Most on-call rotations suffer from three.

Pages that are not actionable. A page is a demand that a human get out of bed and do something. If the alert has no action attached — "CPU is high," "error rate elevated on a service nobody owns anymore" — the responder acknowledges it and goes back to sleep. Every one of these trains the team that pages are ignorable. Audit your last 30 days of pages and count how many resulted in a human taking an action that a machine could not have taken. In teams that have never done this audit, it is common to find that only 20 to 40 percent of pages were genuinely actionable.

Duplicate storms. One database failover fires the database-down alert, plus latency alerts on eight services that call it, plus error-budget alerts on four SLOs, plus a synthetic check per region. That is 40 notifications for one cause. The on-call engineer spends the first fifteen minutes of the incident acknowledging noise instead of diagnosing, and the actual signal — the database alert — is buried in the middle of the storm.

Severity inflation. Every team wants its service taken seriously, so everything becomes critical. When 90 percent of alerts are critical, the word means nothing and routing decisions get made by humans at 3 a.m. instead of by configuration. Severity inflation is a governance failure that no amount of Alertmanager tuning fixes on its own — you need a rubric that is enforced in code review.

The fix for all three is the same shape: decide severity by rubric, route by severity, collapse duplicates by cause, and enrich every alert with enough context that triage takes seconds instead of minutes.

Layer 1: severity-based routing

A severity rubric that survives arguments

Three tiers are enough. More than three and people stop being able to tell them apart.

The one rule that matters: every page must be actionable and urgent. Both. Actionable but not urgent is a ticket. Urgent-looking but not actionable ("load is high") is either info or a broken alert that should be rewritten against a symptom users feel. If it can wait until morning, it is a ticket — this single sentence, enforced ruthlessly, eliminates the majority of overnight noise for most teams.

Severity gets set as a label on the alerting rule itself, which makes it reviewable in the same pull request as the threshold:

groups:
  - name: api-slo
    rules:
      - alert: APIHighErrorRate
        expr: |
          sum(rate(http_requests_total{job="api", code=~"5.."}[5m]))
            / sum(rate(http_requests_total{job="api"}[5m])) > 0.05
        for: 5m
        labels:
          severity: page
          team: platform
        annotations:
          summary: "API 5xx ratio above 5% for 5 minutes"
          runbook_url: "https://runbooks.internal.example.com/api/high-error-rate"

Routing it in Alertmanager

The route tree turns the rubric into behavior. Timing parameters differ per severity on purpose: pages notify fast and re-notify while unresolved; tickets batch up and repeat rarely.

route:
  receiver: ticket-queue
  group_by: ['alertname', 'cluster', 'service']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 12h
  routes:
    - matchers:
        - severity="page"
      receiver: pagerduty-oncall
      group_wait: 30s
      group_interval: 5m
      repeat_interval: 4h
    - matchers:
        - severity="ticket"
      receiver: ticket-queue
      group_wait: 10m
      group_interval: 30m
      repeat_interval: 24h
    - matchers:
        - severity="info"
      receiver: blackhole

receivers:
  - name: pagerduty-oncall
    pagerduty_configs:
      - routing_key_file: /etc/alertmanager/secrets/pagerduty-key
  - name: ticket-queue
    slack_configs:
      - api_url_file: /etc/alertmanager/secrets/slack-webhook
        channel: '#alerts-triage'
        send_resolved: true
  - name: blackhole

Notes on the choices:

• group_wait: 30s on the page route means Alertmanager holds the first notification for 30 seconds to batch alerts arriving together — long enough to collapse a burst into one page, short enough not to delay response meaningfully.
• repeat_interval: 4h on pages means an unresolved page re-fires every four hours. For tickets it is 24 hours, because re-notifying about a non-urgent thing every hour is how channels get muted.
• The blackhole receiver has no configured integrations, which is valid Alertmanager config: alerts routed there are silently dropped but remain queryable in the Alertmanager UI and API. Info-level alerts exist for correlation and debugging, not notification.
• The fallback receiver is the ticket queue, not the pager. An unrouted alert is by definition one nobody made a severity decision about, and undecided must never mean "wake someone."

Layer 2: deduplication and correlation

Routing decides who hears about an alert. This layer decides how many alerts they hear about.

Grouping

group_by above already does the first pass: alerts sharing alertname, cluster, and service collapse into one notification. Tune the key to your topology. Grouping by ['alertname'] alone is too aggressive — a latency alert on two unrelated services becomes one confusing notification. Grouping by every label including instance is too loose — a 50-node rollout failure becomes 50 pages. The right key groups alerts a single responder would treat as one problem.

Inhibition: suppress the downstream

Inhibition rules encode causality: when the cause-level alert fires, symptom-level alerts that share identifying labels are suppressed. The classic case is a dead node — once you know the node is down, every alert about processes on that node is redundant.

inhibit_rules:
  - source_matchers:
      - alertname="NodeDown"
    target_matchers:
      - severity=~"page|ticket"
    equal: ['cluster', 'node']
  - source_matchers:
      - alertname="DatacenterNetworkPartition"
    target_matchers:
      - severity=~"page|ticket"
    equal: ['cluster']
  - source_matchers:
      - severity="page"
    target_matchers:
      - severity="ticket"
    equal: ['alertname', 'cluster', 'service']

The first rule says: while NodeDown fires for a given cluster and node, suppress every page or ticket carrying the same cluster and node labels. The equal list is what keeps inhibition scoped — without it, one dead node in staging would silence production. The third rule handles multi-severity alerts: if you define both a page-level and ticket-level threshold for the same condition, the page inhibits the ticket so you are not notified twice.

Inhibition only works if your alerts carry consistent labels. If half your rules label the node node and half label it instance, the equal matching silently fails. Label hygiene is boring and it is the entire foundation of this layer.

Flap suppression

A metric oscillating around a threshold generates a fire-resolve-fire cycle that is worse than a steady alert, because each cycle is a fresh notification. Two Prometheus-side controls fix most flapping:

• for requires the condition to hold continuously before the alert fires. for: 5m means a 30-second CPU spike never becomes an alert at all. Almost every rule should have a for clause; the ones that should not (hard down, data loss) are the exception.
• keep_firing_for (Prometheus 2.42 and later) is the mirror image: the alert stays firing for that duration after the condition clears, so a metric bouncing across the threshold produces one continuous alert instead of a resolve-refire storm.

      - alert: QueueDepthHigh
        expr: max_over_time(queue_depth{job="worker"}[5m]) > 10000
        for: 10m
        keep_firing_for: 5m
        labels:
          severity: ticket
          team: pipelines

Using max_over_time or avg_over_time in the expression itself is a third smoothing lever when the raw series is spiky by nature.

Layer 3: auto-triage patterns

The first two layers reduce volume. This layer reduces the cost of each alert that remains, which is where acknowledge-to-diagnosis time lives — the stage of the incident lifecycle we break down in how to reduce MTTR.

Every alert carries its runbook. A runbook_url annotation on every alerting rule means the responder clicks one link and lands on the diagnosis steps instead of searching a wiki at 3 a.m. Enforce it mechanically — a CI check that fails any alerting rule missing runbook_url costs an afternoon to build and pays for itself the first weekend. If you do not have runbooks to point at yet, start with DIY incident response runbooks; a runbook that only covers the top five alerts by frequency already covers most pages.

Every alert carries an owner. A team label on every rule does two things: it routes tickets to the right queue via child routes matching on team, and it makes orphaned alerts visible. An alert whose team maps to a squad that was reorganized away eighteen months ago is a deletion candidate, not a routing problem.

Auto-resolve, and treat non-resolving alerts as bugs. Alertmanager marks an alert resolved when Prometheus stops sending it, and send_resolved: true closes the loop in the destination. If an alert routinely stays firing after the underlying issue is fixed, its expression is wrong — usually an absent-metric problem or a threshold measured on a counter that never resets. Alerts that need manual cleanup accumulate as ambient noise in every dashboard.

Correlation into incidents. Grouping and inhibition operate on label matching within one Alertmanager. Correlation goes further: recognizing that 40 alerts across Prometheus, CloudWatch, and your synthetics provider are one incident with one cause. The heuristic for when collapsing is correct: the alerts started within one propagation window of each other (typically 1 to 5 minutes), they share a dependency path — the failing services actually call each other or share infrastructure — and one alert sits topologically upstream of the rest. When those hold, the responder should see one incident titled by the upstream cause, with the 39 downstream alerts attached as evidence rather than delivered as notifications. You can approximate this with disciplined group_by and inhibition inside one stack; across multiple monitoring tools it requires either an event-correlation layer or a human doing the correlation in their head at 3 a.m., which is exactly the cognitive load this system exists to remove.

The honest part: tuning is never done

Every threshold you set decays. Traffic grows, so the request-rate alert that meant "something is wrong" now fires every Tuesday peak. A service gets faster, so the latency threshold that was tight is now so loose it will never fire before users notice. Deploys change baseline memory. The alert set that was clean in January is noisy by June, not because anyone did anything wrong but because the system underneath it moved.

Treat tuning as scheduled work, not reaction. A monthly 30-minute review of three lists keeps the decay in check: the top ten alerts by firing count (candidates for re-thresholding or deletion), every page that was acknowledged without action (candidates for demotion to ticket), and every incident that had no preceding page (candidates for a new alert). Teams that run this loop typically cut page volume by half within a quarter or two; teams that do not run it drift back to noise within about the same period.

And keep the deeper point in view: alert noise reduction treats symptoms. If the same alert fires every week, the durable fix is not a smarter threshold — it is fixing the cause of the recurrence. A triage system buys you the calm to do that engineering; it is not a substitute for it. On-call burnout is rarely caused by the worst incident of the quarter. It is caused by the two hundred small interruptions between incidents, and the only permanent fix for those is making the underlying failures stop happening.

Rules of thumb

• Every page must be actionable and urgent. Fail either test and it is a ticket or info.
• If it can wait until morning, it is a ticket. No exceptions for "but it might get worse" — write a page-severity rule for the actually-bad condition instead.
• Never route unclassified alerts to the pager. Undecided severity defaults to the ticket queue.
• No alerting rule merges without severity, team, and runbook_url. Enforce in CI.
• Acknowledged-without-action is a signal. Two in a row for the same alert means demote or delete it.
• One cause should produce one notification. If a single failure pages you three times, you owe yourself a grouping key or an inhibition rule.
• Deleting an alert is a valid tuning outcome. An alert nobody has acted on in six months is not a safety net; it is noise with a YAML file.

Where this stops scaling by hand

Everything above is buildable with Alertmanager and discipline. The part that stays hard is correlation across tools — recognizing that CloudWatch, Prometheus, and your APM are describing one incident — and keeping severity honest as systems change. That is what CloudThinker's Deep Response Engine automates: Pulse anomaly detection catches drift before thresholds fire, and event correlation collapses cross-tool alert storms into a single incident with the probable cause ranked on top. You can try it on the free tier — 100 premium credits, no card required.