DIY Prometheus Alerting: Rules Files, Alertmanager, and promtool

A shared Postgres instance fails over at 3:10 a.m. Within a minute, #alerts has forty messages: HighErrorRate for checkout, HighErrorRate for billing, HighLatency for search, a TargetDown for the database exporter, and thirty-six more, each one its own notification and each one paging someone. The on-call spends the first ten minutes acknowledging instead of reading.

The difference between that night and a quiet one is often about fifteen lines of Alertmanager YAML:

route:
  receiver: slack-default
  group_by: ['alertname', 'service']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  routes:
    - matchers:
        - severity="page"
      receiver: pagerduty-oncall

A team with this routing tree gets one grouped page per real incident. A team without it gets forty individual Slack messages for the same cascading failure. This guide builds a working Prometheus alerting setup end to end with only native tooling (rules files, Alertmanager, promtool and amtool) and finishes with an honest account of where the native stack stops.

In part one of this series we covered what good alerting looks like: symptom-based rules, the for: duration, scrape-target health, and the PromQL patterns behind most reliable alerts. This is part two, the build. Part three covers what happens after an alert fires.

01Define

Rules files

Alerting and recording rules in YAML, evaluated on a group interval.

02Test

promtool

Syntax checks and unit tests with synthetic series, in CI.

03Route

Alertmanager

Grouping, routing and inhibition decide who hears what, and how often.

04Operate

amtool

Silences for maintenance, route tests before deploys.

05React

Webhooks

POST the alert payload to a bot, a ticketing system or a script.

The full native pipeline this guide builds, from a rule definition to a webhook handler.

Why the defaults don't hold up

Prometheus and Alertmanager ship with sensible mechanics and almost no opinions. Out of the box there are no rules, one receiver, and grouping that only works if you tell it which labels mean "the same problem". Every decision that makes alerting humane is a config decision someone has to make, test and keep current.

That's the job of the six steps below. Rules decide what is wrong. Recording rules keep those decisions cheap. promtool proves the rules fire when you think they will. Alertmanager decides who hears about it. amtool handles the moments you need it to stay quiet. Webhooks are the escape hatch for anything else.

Step 1: Write the alerting rules file

Alerting rules live in YAML files referenced by rule_files: in prometheus.yml, organized into groups that evaluate on a shared interval (alerting rules docs). Here's a production-shaped example covering the two rules every setup needs, scrape-target health and an error-rate symptom:

# /etc/prometheus/rules/availability.yml
groups:
  - name: availability
    rules:
      - alert: TargetDown
        expr: up == 0
        for: 3m
        labels:
          severity: page
        annotations:
          summary: '{{ $labels.job }} target {{ $labels.instance }} is down'
          description: 'Prometheus has failed to scrape {{ $labels.instance }} for 3 minutes. Alerting on this job is blind until it recovers.'

      - alert: HighErrorRate
        expr: |
          sum(rate(http_requests_total{status=~"5.."}[5m])) by (service)
            /
          sum(rate(http_requests_total[5m])) by (service) > 0.05
        for: 10m
        labels:
          severity: page
          team: platform
        annotations:
          summary: '{{ $labels.service }} 5xx ratio is {{ $value | humanizePercentage }}'
          runbook: 'https://runbooks.internal/high-error-rate'

The parts that do the work:

for: 3m. The expression must stay true for three minutes before the alert fires. This single line removes most flapping. During the wait the alert sits in pending state, visible at Status → Rules (or Alerts) in the Prometheus UI.
labels. These are routing keys. Alertmanager's matchers operate on them, so severity and team here decide who gets woken up. Keep the label set small and consistent across every rules file.
annotations with templating. The $labels and $value references in the summary are Go templates evaluated at fire time. Pipe $value through humanize, humanizePercentage or humanizeDuration so pages read "5.2%" instead of "0.052341". Annotations are for people; labels are for machines. Don't put high-cardinality values in labels or you'll multiply alert instances.

Then reference the file and reload:

# prometheus.yml
# rule_files:
#   - /etc/prometheus/rules/*.yml

curl -X POST http://localhost:9090/-/reload   # requires --web.enable-lifecycle

Without --web.enable-lifecycle, send SIGHUP to the Prometheus process instead.

Step 2: Add recording rules so alert queries stay cheap

That HighErrorRate expression runs two sum(rate(...)) aggregations across every series in http_requests_total, at every evaluation interval, forever. On a busy Prometheus this is exactly the kind of query that slowly eats your evaluation budget. Recording rules precompute it once and store the result as a new, cheap series:

groups:
  - name: recording
    interval: 30s
    rules:
      - record: service:http_error_ratio:rate5m
        expr: |
          sum(rate(http_requests_total{status=~"5.."}[5m])) by (service)
            /
          sum(rate(http_requests_total[5m])) by (service)

The naming convention is level:metric:operations: aggregation level, metric name, transformation. Your alert rule then collapses to:

- alert: HighErrorRate
  expr: service:http_error_ratio:rate5m > 0.05
  for: 10m

Two wins. The alert evaluation is now a trivial lookup, and the same precomputed series feeds your Grafana panels, so the dashboard you stare at during an incident shows exactly the number the alert fired on, not a subtly different ad-hoc query.

To see whether your rule groups are keeping up at all, compare evaluation time to the interval:

prometheus_rule_group_last_duration_seconds
  / prometheus_rule_group_interval_seconds > 0.5

Any group above 0.5 is spending more than half its interval evaluating, and is a candidate for recording rules.

Step 3: Test rules with promtool before they page anyone

Every rules change goes through two gates. First, syntax:

promtool check rules /etc/prometheus/rules/*.yml
promtool check config /etc/prometheus/prometheus.yml

Second, and this is the step most teams skip, unit tests. You feed synthetic series in and assert which alerts fire and when:

# tests/availability_test.yml
rule_files:
  - ../rules/availability.yml

evaluation_interval: 1m

tests:
  - interval: 1m
    input_series:
      - series: 'up{job="api", instance="10.0.1.5:9090"}'
        values: '1 1 0 0 0 0 0'
    alert_rule_test:
      # at 2m the target just went down, inside the for: window, so no alert
      - eval_time: 2m
        alertname: TargetDown
        exp_alerts: []
      # at 6m it has been down for 3m+, so the alert must fire
      - eval_time: 6m
        alertname: TargetDown
        exp_alerts:
          - exp_labels:
              severity: page
              job: api
              instance: 10.0.1.5:9090
            exp_annotations:
              summary: 'api target 10.0.1.5:9090 is down'
              description: 'Prometheus has failed to scrape 10.0.1.5:9090 for 3 minutes. Alerting on this job is blind until it recovers.'

promtool test rules tests/availability_test.yml

Wire both commands into CI on the rules repo. A wrong threshold caught by promtool test rules costs a code review comment. Caught in production, it costs a 3 a.m. page, or worse, a silent miss.

Step 4: Configure Alertmanager: routing, grouping, inhibition

Alertmanager takes firing alerts from Prometheus and decides who hears about them and how often (configuration docs). Here's the full config the opening snippet came from:

# /etc/alertmanager/alertmanager.yml
route:
  receiver: slack-default
  group_by: ['alertname', 'service']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  routes:
    - matchers:
        - severity="page"
      receiver: pagerduty-oncall
    - matchers:
        - team="data"
      receiver: slack-data

receivers:
  - name: slack-default
    slack_configs:
      - api_url_file: /etc/alertmanager/secrets/slack_url
        channel: '#alerts'
        send_resolved: true
  - name: pagerduty-oncall
    pagerduty_configs:
      - routing_key_file: /etc/alertmanager/secrets/pd_key
  - name: slack-data
    slack_configs:
      - api_url_file: /etc/alertmanager/secrets/slack_url
        channel: '#data-alerts'

inhibit_rules:
  - source_matchers:
      - severity="critical"
    target_matchers:
      - severity="warning"
    equal: ['alertname', 'service']

The knobs worth understanding:

group_by collapses alerts sharing those label values into one notification. Twenty pods failing in one service becomes one message listing twenty instances.
group_wait: 30s buffers the first notification so alerts arriving seconds apart join the same group. group_interval: 5m rate-limits updates about new alerts joining an existing group. repeat_interval: 4h controls re-notification for alerts still firing. Set it too low and you train people to ignore repeats.
inhibit_rules suppress a lower-severity alert while a matching higher-severity one fires on the same equal labels. When the critical version of an alert is already paging, the warning version stays quiet.

Go back to the 3:10 a.m. failover. With group_by: ['alertname', 'service'], the forty messages become one per alert name per service. Add the TargetDown inhibition from part one, and the warnings for instances whose exporter is down disappear too. That's most of the storm gone, without losing a single alert from the UI.

Verify the routing tree does what you think before deploying:

amtool check-config /etc/alertmanager/alertmanager.yml

amtool config routes test \
  --config.file=/etc/alertmanager/alertmanager.yml \
  severity=page service=checkout
# => pagerduty-oncall

Step 5: Silences with amtool

During planned maintenance, silence alerts. Don't delete or comment out rules you'll forget to restore:

# Silence one alert on one service for 2 hours
amtool silence add alertname=HighErrorRate service=checkout \
  --alertmanager.url=http://localhost:9093 \
  --duration=2h --author=steve \
  --comment="checkout DB failover, CHG-4112"

# What's currently silenced, and what's currently firing
amtool silence query --alertmanager.url=http://localhost:9093
amtool alert query --alertmanager.url=http://localhost:9093

# End it early
amtool silence expire SILENCE_ID --alertmanager.url=http://localhost:9093

Silences expire on their own. The failure mode is a --duration=24h silence for a 2-hour maintenance window that swallows a real incident at hour six. Keep silences short and scoped to the narrowest matcher set possible. For maintenance that repeats on a schedule, Alertmanager's time_intervals with mute_time_intervals on a route does the same job without anyone remembering to run amtool.

Step 6: Webhook receivers, the native automation escape hatch

The webhook_configs receiver POSTs a JSON payload (group labels, common annotations, and the full alert list with status, labels, annotations and startsAt) to any HTTP endpoint (webhook docs):

- name: webhook-automation
  webhook_configs:
    - url: http://alert-bot.internal:8080/hook
      send_resolved: true

Teams wire this to ticket creation, chat bots that post enriched context, or scripts that restart a known-flaky service. It works, and for one or two well-understood alerts it's worth doing. But be honest about what you're signing up for: each webhook handler is a small service you now own, with its own deploys and failure modes, and with the hardcoded assumption that this alert always has that cause. Most webhook automation ossifies into a script nobody trusts enough to extend.

What each native tool covers

Tool	Solves	Leaves to a person
Alerting rules with `for:`	Firing on sustained conditions, not blips	Whether the condition is the real problem
Recording rules	Cheap, consistent evaluation	Which other series explain the change
`promtool test rules`	Rules fire when and how you expect	Whether the threshold still matches production
Alertmanager routing and inhibition	One grouped notification to the right team	What caused it
`amtool` silences	Quiet during known work	Remembering to scope and expire them
Webhooks	A fixed reaction to a fixed alert	Deciding if that reaction fits this time

The ceiling: routing manages notifications, not incidents

Run all six steps and you have a solid setup: tested rules, cheap queries, a routing tree that groups and deduplicates, silences with an audit trail. That's better than most shops. Now look at what the stack still doesn't do.

01Native

Grouping

Twenty failing pods arrive as one message instead of twenty.

02Native

Inhibition

The cause pages; its consequences stay quiet.

03Native

Silences

Known maintenance stops paging without editing rules.

04Still manual

Investigation

Which instance, which change, which dependency: the queries a person runs after the page.

Everything Alertmanager does decides how alerts are delivered. None of it explains why one fired.

Everything above manages the delivery of alerts. Grouping, inhibition, repeat_interval, silences: these decide how many notifications you receive and where. None of them touch what happens after the page lands.

The investigation is still entirely manual. When HighErrorRate fires on checkout, a person opens Grafana and runs the follow-up PromQL. Is up also down? Did latency move? What deployed in the last hour? Then they correlate across dashboards. Alertmanager knows none of this. It delivered a message.

Webhooks automate reactions, not reasoning. A webhook can restart a service. It can't tell whether restarting is the right call this time.

The config itself becomes an artifact to maintain. Routing trees grow routes nobody remembers; inhibition rules mask alerts in ways nobody documents. Six months in, the YAML is its own source of incidents.

The native stack gets an alert to the right person efficiently. The 30 to 60 minutes of triage that follow every page are the part nothing in this article shortens.

What comes next

The follow-up queries you run after every page (correlating the symptom across related series, checking scrape health, comparing against the last deploy window) are mechanical enough to automate. CloudThinker connects read-only to your Prometheus and Alertmanager APIs, picks up firing alerts, runs that investigation, and presents the likely cause with evidence. Remediation stays gated behind your approval. That's part three of this series.