Prometheus Alerting: 6 Failure Modes That Bury Real Incidents

Last Tuesday your latency alert fired, correctly, at 2:14 a.m. The on-call engineer opened Grafana, then the Prometheus expression browser, then Grafana again. Forty minutes later they found the actual cause: a node exporter that had been dead for nine days. Nothing ever alerted on it. The target went down, its metrics went stale, and every threshold rule that depended on those metrics simply stopped evaluating to anything at all.

That incident contains most of what goes wrong with Prometheus alerting. The rules you wrote fire loudly on symptoms while the causes rot silently underneath. The rules you didn't write (scrape health, missing data) never fire at all. And Alertmanager forwards the resulting mess to Slack one notification at a time until everyone mutes the channel.

The failure modes are predictable. This guide covers the six that account for almost every noisy, blind or misleading Prometheus setup: what each one looks like, why it happens, one concrete PromQL or config example, and what fixing it typically buys you.

This is part one of a three-part series. Part two is a hands-on guide to alert automation with the native Prometheus stack: rule files, Alertmanager routing, promtool tests. Part three covers putting an AI agent on the receiving end of your alerts.

01Noise

Paging on causes

CPU and disk rules wake people while users feel nothing.

02Noise

No for: clause

One bad evaluation fires a page that resolves thirty seconds later.

03Blindness

No up == 0 rule

A dead exporter takes every rule that depends on it down silently.

04Blindness

Cardinality

An unbounded label slows rule evaluation until alerts fire late or not at all.

05Noise

Default Alertmanager

One node failure arrives as forty separate notifications.

06Correctness

PromQL misuse

Raw counters, averaged percentiles and vanishing series fire on arithmetic, not reality.

Three failure modes make alerting noisy, two make it blind, and one makes it wrong.

Why Prometheus rule sets drift toward noise

Prometheus doesn't have a noise problem by design. It has one by accumulation, and the reasons are structural.

Rules are written per service, not per system. Each team adds the alerts that make sense for its own service on launch day. Nobody owns the question "what does the on-call see when the shared database fails?", so the answer is "forty things at once".

The easy metrics are the wrong ones. node_exporter gives you CPU, memory and disk for free on day one. A service's error ratio needs instrumentation and an SLO conversation. Easy alerts accumulate; important ones wait.

Silence looks like health. A rule that can't evaluate (no data, a dead target, a renamed metric) doesn't fail loudly. It just never fires, which on a dashboard looks exactly like a healthy service.

Routing is a separate file nobody reviews. Alertmanager config works badly by default rather than failing, so it never makes it up the backlog.

The six failure modes below are what those four forces produce.

1. Paging on causes instead of symptoms

What it is. Rules that page a human because CPU crossed 80%, disk crossed 70%, or a queue has "too many" items, while users are entirely unaffected. Meanwhile there's often no page at all for what users actually experience: error ratio and latency.

Why it happens. Cause-based rules are easy to write on day one. node_cpu_seconds_total is right there; your service's error ratio needs instrumentation and a real SLO conversation. So the easy alerts accumulate and the important ones never get written. The Google SRE book's monitoring chapter has made this argument since 2016: page on symptoms, use causes for debugging context.

What good looks like. One symptom-based page per service, expressed as a ratio of counters:

- alert: HighErrorRatio
  expr: |
    sum by (job) (rate(http_requests_total{status=~"5.."}[5m]))
      /
    sum by (job) (rate(http_requests_total[5m])) > 0.05
  for: 10m
  labels:
    severity: page
  annotations:
    summary: '5xx ratio above 5% for 10 minutes on {{ $labels.job }}'

Demote the CPU and disk rules to severity: ticket. They're context, not emergencies.

Typical impact. Teams that move pages from causes to symptoms typically cut page volume by 40 to 70% while catching more user-facing incidents, because a symptom rule fires no matter which of a hundred causes produced it.

2. Zero-delay alerts flap

What it is. An alerting rule with no for: clause fires the instant its expression is true for a single evaluation: one garbage-collection pause, one slow scrape, one retried batch job. Thirty seconds later it resolves. Then it fires again. PagerDuty calls this pattern flapping; your on-call rotation calls it something less printable.

Why it happens. for: is optional, and the cost of omitting it isn't visible in testing. It only shows up as noise in production, weeks later, one spike at a time.

How to fix it. Give every paging rule a for: duration that the condition must hold continuously:

- alert: HighErrorRatio
  expr: sum by (job) (rate(http_requests_total{status=~"5.."}[5m])) / sum by (job) (rate(http_requests_total[5m])) > 0.05
  for: 10m

Rule of thumb: for: should be at least two evaluation intervals, so a single bad scrape can't page anyone, and for most pages 5 to 15 minutes is right. Yes, this adds detection latency. That's the deliberate trade: an incident that self-heals in four minutes didn't need a human awake at 2 a.m. The alerting rules documentation covers the mechanics, including keep_firing_for for the mirror-image problem of alerts that resolve and re-fire during a slow recovery.

Typical impact. Adding sensible for: durations to a rule set that lacks them commonly eliminates half or more of all firings, the transient half nobody should have seen.

3. The silent failure: nobody alerts on `up == 0`

What it is. For every scrape target, Prometheus synthesizes an up metric: 1 if the scrape succeeded, 0 if it failed. When an exporter dies, up goes to 0 and every other series from that target goes stale within about five minutes. That's the dangerous part. Your threshold alerts on those metrics don't fire. They can't: there's no data to compare against a threshold. The monitoring gap is itself invisible to the monitoring.

01Day 0

Exporter dies

The scrape fails. Prometheus records up = 0 for the target.

02Minute 5

Series go stale

Every other series from that target stops returning samples.

03Rules

Evaluate to nothing

Threshold rules have no data to compare, so they can never fire.

04Day 9

Found by accident

Someone notices during an unrelated incident.

The only place this failure is visible is the up metric. If nothing alerts on it, the gap is invisible to the monitoring itself.

Why it happens. up doesn't belong to any service team, so nobody writes the rule. And the failure is quiet by nature: the graphs don't go red, they go flat or empty, and an empty panel at 2 a.m. reads as "nothing happening".

How to detect it. Right now, run this in the Prometheus expression browser (or check Status → Targets in the UI):

up == 0

Then make it permanent:

- alert: TargetDown
  expr: up == 0
  for: 5m
  labels:
    severity: page
  annotations:
    summary: '{{ $labels.job }} target {{ $labels.instance }} unreachable for 5 minutes'

Typical impact. This is a blindness problem, not a noise problem. In audits of real-world Prometheus setups it's common to find several percent of targets down, some for weeks, exactly like the node exporter in the opening story.

4. Label cardinality pressure

What it is. Every unique combination of label values is a separate time series. Put a user ID, raw URL path or request UUID into a label and one metric becomes a million series. Prometheus memory climbs, queries slow down, and (the alerting-specific consequence) rule groups start missing their evaluation deadlines. Alerts then fire late or intermittently for reasons that look like ghosts.

Why it happens. A developer adds path as a label because it seems useful, and it is, right up until someone runs a URL scanner against the service and mints 400,000 series in an afternoon.

How to detect it. Check Status → TSDB Status in the Prometheus UI for the highest-cardinality metric names and label pairs, or query for the worst offenders:

topk(10, count by (__name__)({__name__=~".+"}))

That query is expensive, so run it on a quiet Prometheus, not in a dashboard refresh loop. Watch prometheus_tsdb_head_series over time for step changes, and prometheus_rule_group_iterations_missed_total for the proof that cardinality is already hurting rule evaluation:

increase(prometheus_rule_group_iterations_missed_total[1h]) > 0

Typical impact. A single unbounded label can multiply a metric's series count 100x or more. Cleaning up the top two or three offenders often halves head-series count and puts rule evaluation back on schedule.

5. Alertmanager left on defaults

What it is. Prometheus decides what is wrong; Alertmanager decides who hears about it and how often. A near-default Alertmanager config sends every alert to one receiver with generic grouping. When a node dies and 40 rules fire, on-call gets a wall of notifications where the one that matters (the node is down) is buried under 39 consequences of it.

Why it happens. Alertmanager configuration is a separate file and a separate mental model (routes, grouping, inhibition), and it works badly by default rather than failing loudly. So it never makes it up the backlog.

What good looks like. Group related alerts into single notifications, and inhibit the consequences when the cause is already firing:

route:
  receiver: team-default
  group_by: ['alertname', 'cluster']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h

inhibit_rules:
  - source_matchers:
      - alertname = TargetDown
    target_matchers:
      - severity = warning
    equal: ['instance']

With that inhibition rule, the moment TargetDown fires for an instance, every warning-level alert for the same instance is suppressed. You get one notification about the cause instead of a storm about the effects.

Typical impact. Grouping alone commonly collapses notification volume 5 to 10x during real incidents; inhibition removes most of what remains.

6. Most good alerts use the same three PromQL patterns

What it is. The long tail of bad rules (raw counter comparisons, averaged percentiles, thresholds on gauges that vanish) mostly comes from not knowing three PromQL patterns. These are the PromQL examples worth memorizing.

rate() on counters, always. A counter's absolute value is meaningless because it resets on restart; its rate is the signal. Never alert on http_requests_total > 1000000. Alert on the rate over a window at least 4x your scrape interval:

sum by (job) (rate(http_requests_total{status=~"5.."}[5m])) > 5

absent() for data that should exist. A threshold rule on a metric that disappears evaluates to nothing and never fires. If a job's heartbeat matters, alert on its absence explicitly:

absent(up{job="billing"})

This catches the case section 3 doesn't: the job vanished from service discovery entirely, so there's no up series left to equal zero.

histogram_quantile() for latency. Never average percentiles across instances; it's mathematically wrong. Aggregate the histogram buckets first, then take the quantile:

histogram_quantile(0.99,
  sum by (le) (rate(http_request_duration_seconds_bucket[5m]))
) > 0.5

The PromQL functions documentation covers the sharp edges of all three.

Typical impact. These patterns are the difference between rules that fire on real conditions and rules that fire on arithmetic artifacts. Most rule-set audits find 20 to 40% of existing rules misusing counters or percentiles.

The six failure modes at a glance

Failure mode	Symptom on call	Quick check	Fix
Paging on causes	Pages with nothing for users to feel	Count `severity: page` rules on node metrics	Symptom-based ratio rules; demote causes to tickets
No `for:`	Alerts that resolve within a minute	Rules without `for:` in `/api/v1/rules`	`for:` of 5 to 15 minutes on pages
No `up == 0` rule	Flat graphs, missed incidents	`up == 0` in the expression browser	`TargetDown` plus `absent()` for critical jobs
Cardinality	Late or intermittent firings	Status → TSDB Status	Drop or bound the offending labels
Default Alertmanager	Notification storms	One receiver, no `inhibit_rules`	Grouping, routes, inhibition
PromQL misuse	Alerts that fire on artifacts	Raw counters, averaged quantiles	`rate()`, `absent()`, `histogram_quantile()`

Alert tuning never stays done

"We cleaned up our alerts last quarter" is always a temporary statement, because every one of these failure modes regenerates. Each new service ships with cause-based rules and no for:. Each new exporter is a new target nobody watches up for. Each new label is a cardinality bet. Each new team bolts another receiver onto Alertmanager without touching grouping.

And even a perfectly tuned rule set only solves detection. When HighErrorRatio fires at 2 a.m., someone still has to run the next five PromQL queries, correlate the symptom with a cause and decide what to do. That's exactly the forty minutes the engineer in the opening story lost.

Start by fixing the rules. Part two of this series walks through the full DIY toolkit: rule files, promtool unit tests, Alertmanager routing and silences. But know that the investigation step is where the real time goes.

When the alert fires, then what?

Detection is half the job. CloudThinker connects to your Prometheus and Alertmanager APIs read-only, picks up firing alerts, and runs the follow-up queries a person would: correlating the symptom across related series, checking scrape health, comparing against recent changes. Then it presents a likely cause with the evidence. Remediation stays gated behind your approval. Part three walks through one alert end to end.