DIY Prometheus Alerting: Rules Files, Alertmanager, and promtool

The difference between an on-call rotation that sleeps and one that doesn't is often about fifteen lines of Alertmanager YAML:

route:
  receiver: slack-default
  group_by: ['alertname', 'service']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  routes:
    - matchers:
        - severity="page"
      receiver: pagerduty-oncall

A team with this routing tree gets one grouped page per real incident. A team without it gets forty individual Slack messages for the same cascading failure. This guide builds a working Prometheus alerting setup end to end with only native tooling — rules files, Alertmanager, promtool, and amtool — and finishes with an honest accounting of where the native stack stops.

In part one of this series we covered what good alerting looks like: symptom-based rules, the for: duration, scrape-target health, and the PromQL patterns behind most reliable alerts. This is part two — the build. Part three covers what happens after an alert fires.

Step 1: Write the alerting rules file

Alerting rules live in YAML files referenced by rule_files: in prometheus.yml, organized into groups that evaluate on a shared interval (alerting rules docs). A production-shaped example covering the two rules every setup needs — scrape-target health and an error-rate symptom:

# /etc/prometheus/rules/availability.yml
groups:
  - name: availability
    rules:
      - alert: TargetDown
        expr: up == 0
        for: 3m
        labels:
          severity: page
        annotations:
          summary: '{{ $labels.job }} target {{ $labels.instance }} is down'
          description: 'Prometheus has failed to scrape {{ $labels.instance }} for 3 minutes. Alerting on this job is blind until it recovers.'

      - alert: HighErrorRate
        expr: |
          sum(rate(http_requests_total{status=~"5.."}[5m])) by (service)
            /
          sum(rate(http_requests_total[5m])) by (service) > 0.05
        for: 10m
        labels:
          severity: page
          team: platform
        annotations:
          summary: '{{ $labels.service }} 5xx ratio is {{ $value | humanizePercentage }}'
          runbook: 'https://runbooks.internal/high-error-rate'

The parts that do the work:

• for: 3m — the expression must stay true for three minutes before the alert fires. This single line eliminates most flapping. During the wait the alert sits in pending state, visible at Status → Rules in the Prometheus UI.
• labels — these are routing keys. Alertmanager's matchers operate on them, so severity and team here determine who gets woken up. Keep the label set small and consistent across every rules file.
• annotations with templating — the $labels and $value references in the summary above are Go templates evaluated at fire time. Pipe $value through humanize, humanizePercentage, or humanizeDuration so pages read "5.2%" instead of "0.052341". Annotations are for humans; labels are for machines. Don't put high-cardinality values in labels or you'll multiply alert instances.

Then reference the file and reload:

# prometheus.yml
# rule_files:
#   - /etc/prometheus/rules/*.yml

curl -X POST http://localhost:9090/-/reload   # requires --web.enable-lifecycle

Step 2: Add recording rules so alert queries stay cheap

That HighErrorRate expression runs two sum(rate(...)) aggregations across every series in http_requests_total — at every evaluation interval, forever. On a busy Prometheus this is exactly the kind of query that slowly eats your evaluation budget. Recording rules precompute it once and store the result as a new, cheap series:

groups:
  - name: recording
    interval: 30s
    rules:
      - record: service:http_error_ratio:rate5m
        expr: |
          sum(rate(http_requests_total{status=~"5.."}[5m])) by (service)
            /
          sum(rate(http_requests_total[5m])) by (service)

The naming convention is level:metric:operations — aggregation level, metric name, transformation. Your alert rule then collapses to:

- alert: HighErrorRate
  expr: service:http_error_ratio:rate5m > 0.05
  for: 10m

Two wins: the alert evaluation is now a trivial lookup, and the same precomputed series feeds your Grafana panels — so the dashboard you stare at during an incident shows exactly the number the alert fired on, not a subtly different ad-hoc query.

Step 3: Test rules with promtool before they page anyone

Every rules change goes through two gates. First, syntax:

promtool check rules /etc/prometheus/rules/*.yml
promtool check config /etc/prometheus/prometheus.yml

Second — and this is the step most teams skip — unit tests. You feed synthetic series in, assert which alerts fire and when:

# tests/availability_test.yml
rule_files:
  - ../rules/availability.yml

evaluation_interval: 1m

tests:
  - interval: 1m
    input_series:
      - series: 'up{job="api", instance="10.0.1.5:9090"}'
        values: '1 1 0 0 0 0 0'
    alert_rule_test:
      # at 2m the target just went down — inside the for: window, no alert
      - eval_time: 2m
        alertname: TargetDown
        exp_alerts: []
      # at 6m it has been down for 3m+ — the alert must fire
      - eval_time: 6m
        alertname: TargetDown
        exp_alerts:
          - exp_labels:
              severity: page
              job: api
              instance: 10.0.1.5:9090
            exp_annotations:
              summary: 'api target 10.0.1.5:9090 is down'
              description: 'Prometheus has failed to scrape 10.0.1.5:9090 for 3 minutes. Alerting on this job is blind until it recovers.'

promtool test rules tests/availability_test.yml

Wire both commands into CI on the rules repo. A wrong threshold caught by promtool test rules costs a code review comment; caught in production, it costs a 3 a.m. page — or worse, a silent miss.

Step 4: Configure Alertmanager — routing, grouping, inhibition

Alertmanager takes firing alerts from Prometheus and decides who hears about them and how often (configuration docs). The full config the opening snippet came from:

# /etc/alertmanager/alertmanager.yml
route:
  receiver: slack-default
  group_by: ['alertname', 'service']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  routes:
    - matchers:
        - severity="page"
      receiver: pagerduty-oncall
    - matchers:
        - team="data"
      receiver: slack-data

receivers:
  - name: slack-default
    slack_configs:
      - api_url_file: /etc/alertmanager/secrets/slack_url
        channel: '#alerts'
        send_resolved: true
  - name: pagerduty-oncall
    pagerduty_configs:
      - routing_key_file: /etc/alertmanager/secrets/pd_key
  - name: slack-data
    slack_configs:
      - api_url_file: /etc/alertmanager/secrets/slack_url
        channel: '#data-alerts'

inhibit_rules:
  - source_matchers:
      - severity="critical"
    target_matchers:
      - severity="warning"
    equal: ['alertname', 'service']

The knobs worth understanding:

• group_by collapses alerts sharing those label values into one notification. Twenty pods failing in one service becomes one message listing twenty instances.
• group_wait: 30s buffers the first notification so alerts arriving seconds apart join the same group. group_interval: 5m rate-limits updates about new alerts joining an existing group. repeat_interval: 4h controls re-notification for still-firing alerts — set it too low and you retrain people to ignore repeats.
• inhibit_rules suppress a lower-severity alert while a matching higher-severity one fires on the same equal labels. When the critical version of an alert is already paging, the warning version stays quiet.

Verify the routing tree does what you think before deploying:

amtool check-config /etc/alertmanager/alertmanager.yml

amtool config routes test \
  --config.file=/etc/alertmanager/alertmanager.yml \
  severity=page service=checkout
# => pagerduty-oncall

Step 5: Silences with amtool

During planned maintenance, silence — don't delete or comment out rules you'll forget to restore:

# Silence one alert on one service for 2 hours
amtool silence add alertname=HighErrorRate service=checkout \
  --alertmanager.url=http://localhost:9093 \
  --duration=2h --author=steve \
  --comment="checkout DB failover, CHG-4112"

# What's currently silenced, and what's currently firing
amtool silence query --alertmanager.url=http://localhost:9093
amtool alert query --alertmanager.url=http://localhost:9093

# End it early
amtool silence expire SILENCE_ID --alertmanager.url=http://localhost:9093

Silences expire on their own — the failure mode is a --duration=24h silence for a 2-hour maintenance window that swallows a real incident at hour six. Keep silences short and scoped to the narrowest matcher set possible.

Step 6: Webhook receivers — the native automation escape hatch

The webhook_configs receiver POSTs a JSON payload — group labels, common annotations, and the full alert list with status, labels, annotations, and startsAt — to any HTTP endpoint (webhook docs):

- name: webhook-automation
  webhook_configs:
    - url: http://alert-bot.internal:8080/hook
      send_resolved: true

Teams wire this to ticket creation, chat bots that post enriched context, or scripts that restart a known-flaky service. It works, and for one or two well-understood alerts it's worth doing. But be honest about what you're signing up for: each webhook handler is a small service you now own — with its own deploys, failure modes, and the hardcoded assumption that this alert always has that cause. Most webhook automation ossifies into a script nobody trusts enough to extend.

The ceiling: routing manages notifications, not incidents

Run all six steps and you have a genuinely solid setup — tested rules, cheap queries, a routing tree that groups and deduplicates, silences with audit trails. That's better than most shops. Now notice what the stack still doesn't do.

Everything above manages the delivery of alerts. Grouping, inhibition, repeat_interval, silences — these decide how many notifications you receive and where. None of them touch what happens after the page lands.

The investigation is still entirely manual. When HighErrorRate fires on checkout, a human opens Grafana, runs the follow-up PromQL — is up also down? did latency move? what deployed in the last hour? — and correlates across dashboards. Alertmanager knows none of this; it delivered a message.

Webhooks automate reactions, not reasoning. A webhook can restart a service; it cannot determine whether restarting is the right call this time.

The config itself becomes an artifact to maintain. Routing trees grow routes nobody remembers; inhibition rules mask alerts in ways nobody documents. Six months in, the YAML is its own source of incidents.

The native stack gets an alert to the right person efficiently. The 30–60 minutes of triage that follow every page — that's the part nothing in this article shortens.

What comes next

The follow-up queries you run after every page — correlating the symptom across related series, checking scrape health, comparing against the last deploy window — are mechanical enough to automate. CloudThinker agents connect read-only to your Prometheus and Alertmanager APIs, pick up firing alerts, run that investigation, and present the likely cause with evidence — remediation stays gated behind your approval. That's part three of this series.

Or see it against your own alerts today: Try CloudThinker free — 100 premium credits, no card required.