Grafana Alerting Automation with Native Tools: A Hands-On Guide

A notification policy can decide who hears about a firing alert, in what channel, how often, and during which hours. What it cannot decide is whether the alert matters, what caused it, or what to do about it. That distinction is the whole shape of this guide to Grafana alerting automation: the native layer for routing and escalation is genuinely capable, and knowing exactly where it ends is what separates a quiet on-call rotation from a noisy one.

In part one of this series we mapped the dashboards-to-action gap: rule sprawl across data sources, noisy versus meaningful alerts, and what a healthy Grafana Alerting setup looks like. This part is the build. We'll implement Grafana alerting automation end to end with native features only — alert rules as code, notification policies, contact points (including webhooks that trigger real actions), silences, mute timings, and Grafana OnCall for escalation. Everything below runs on Grafana 10/11 OSS or Grafana Cloud, with exact provisioning files you can commit today.

Step 1: Alert rules as code, not as click-ops

Rules created in the UI drift, get orphaned, and can't be reviewed. Grafana's file provisioning lets you define every alert rule in YAML under /etc/grafana/provisioning/alerting/ — Grafana loads them at startup, and provisioned rules are read-only in the UI, which is exactly what you want.

A complete rule: query Prometheus, reduce to a threshold expression, hold for five minutes, attach routing labels:

# /etc/grafana/provisioning/alerting/api-latency.yaml
apiVersion: 1
groups:
  - orgId: 1
    name: api-slo
    folder: production
    interval: 1m
    rules:
      - uid: api-p99-latency
        title: API p99 latency above 800ms
        condition: C
        data:
          - refId: A
            relativeTimeRange:
              from: 600
              to: 0
            datasourceUid: prometheus-main
            model:
              expr: histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket{job="api"}[5m])) by (le))
              instant: true
              refId: A
          - refId: C
            datasourceUid: __expr__
            model:
              type: threshold
              expression: A
              conditions:
                - evaluator:
                    type: gt
                    params: [0.8]
              refId: C
        for: 5m
        labels:
          severity: critical
          team: platform
        annotations:
          summary: "API p99 latency is {{ $values.A }}s (threshold 0.8s)"
          runbook_url: https://runbooks.internal/api-latency

Three details carry most of the value. The for: 5m clause suppresses flapping — the condition must hold for five straight minutes before the alert fires. The labels block is not decoration; it's the routing input your notification policies match on, so standardize severity and team across every rule now. And runbook_url in annotations flows through to every notification, including webhook payloads.

Already have rules built in the UI? Export them in provisioning format instead of rewriting them:

curl -s -H "Authorization: Bearer $GRAFANA_SA_TOKEN" \
  "https://grafana.example.com/api/v1/provisioning/alert-rules/export?format=yaml" \
  > alerting-rules.yaml

Commit that file, wire it into your image or config management, and the UI stops being your source of truth (export API docs).

Step 2: Notification policies — the routing tree

Notification policies are a tree: every firing alert enters at the root and descends until a route's matchers accept it. This is where you encode "critical pages someone, warnings go to Slack, everything else stays quiet":

# /etc/grafana/provisioning/alerting/policies.yaml
apiVersion: 1
policies:
  - orgId: 1
    receiver: platform-slack
    group_by: ["alertname", "cluster"]
    group_wait: 30s
    group_interval: 5m
    repeat_interval: 4h
    routes:
      - receiver: oncall-critical
        matchers:
          - severity = critical
        group_wait: 10s
        repeat_interval: 1h
      - receiver: platform-slack
        matchers:
          - team = platform
        mute_time_intervals:
          - weekend-maintenance

The three timers do more for alert fatigue than any other setting in Grafana. group_wait (30s) buffers the initial burst so one cascading failure produces one grouped notification, not forty. group_interval (5m) batches new alerts joining an existing group. repeat_interval (4h) is how often an unresolved alert re-notifies — the default is 4 hours, and teams that lower it to 15 minutes are usually the same teams that stop reading their alert channel by Friday.

Contact points are provisioned the same way:

# /etc/grafana/provisioning/alerting/contact-points.yaml
apiVersion: 1
contactPoints:
  - orgId: 1
    name: platform-slack
    receivers:
      - uid: cp-slack-platform
        type: slack
        settings:
          recipient: "#alerts-platform"
          token: $SLACK_BOT_TOKEN
  - orgId: 1
    name: oncall-critical
    receivers:
      - uid: cp-webhook-oncall
        type: webhook
        settings:
          url: https://oncall.example.com/integrations/v1/grafana/xxxx/
          httpMethod: POST

Step 3: Webhooks — where routing becomes action

The webhook contact point is the native escape hatch from "notify a human" to "trigger a system." Grafana POSTs a JSON body like this for every notification:

{
  "receiver": "queue-scaler",
  "status": "firing",
  "alerts": [
    {
      "status": "firing",
      "labels": {
        "alertname": "HighQueueDepth",
        "severity": "warning",
        "service": "ingest",
        "team": "platform"
      },
      "annotations": {
        "summary": "Ingest queue depth above 10k for 10m",
        "runbook_url": "https://runbooks.internal/ingest-queue"
      },
      "startsAt": "2026-07-11T03:12:44Z",
      "endsAt": "0001-01-01T00:00:00Z",
      "fingerprint": "b0d3f2a1c4e5d6f7",
      "dashboardURL": "https://grafana.example.com/d/ingest-overview",
      "panelURL": "https://grafana.example.com/d/ingest-overview?viewPanel=4",
      "valueString": "[ var='A' labels={service=ingest} value=14213 ]"
    }
  ],
  "commonLabels": { "alertname": "HighQueueDepth", "service": "ingest" },
  "externalURL": "https://grafana.example.com/",
  "title": "[FIRING:1] HighQueueDepth (ingest warning)",
  "state": "alerting"
}

Everything an automation endpoint needs is here: labels for dispatch logic, valueString for the measured value, fingerprint for deduplication, dashboardURL/panelURL for context links. Typical mid-market uses: a Lambda or Cloud Function that scales a consumer group when HighQueueDepth fires, a script that restarts a stuck systemd unit, a ticket-creation endpoint keyed on fingerprint so repeat firings update rather than duplicate.

Be honest about what you're building, though: each webhook target is a bespoke service you now own — with its own auth, retries, idempotency handling, and failure modes. One webhook automation is a quick win. Fifteen of them is an unversioned remediation platform maintained in your spare time.

Step 4: Silences and mute timings — planned quiet

Two different tools for two different kinds of quiet. Mute timings are recurring and provisioned — use them for known windows like nightly batch jobs or weekend maintenance:

# /etc/grafana/provisioning/alerting/mute-timings.yaml
apiVersion: 1
muteTimes:
  - orgId: 1
    name: weekend-maintenance
    time_intervals:
      - weekdays: ["saturday", "sunday"]
        times:
          - start_time: "02:00"
            end_time: "04:00"

Reference the name from any route's mute_time_intervals (as in the policy tree above) and matching alerts go silent during the window — they still fire and record, they just don't notify (docs).

Silences are ad hoc and label-matched — the right tool when you're about to do maintenance on one service tonight. Create one through the Alertmanager-compatible API with a service account token:

curl -s -X POST \
  "https://grafana.example.com/api/alertmanager/grafana/api/v2/silences" \
  -H "Authorization: Bearer $GRAFANA_SA_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "matchers": [
      { "name": "service", "value": "ingest", "isRegex": false, "isEqual": true }
    ],
    "startsAt": "2026-07-12T22:00:00Z",
    "endsAt": "2026-07-13T02:00:00Z",
    "createdBy": "steve",
    "comment": "ingest node pool upgrade, change #4821"
  }'

Silences expire on their own — which is precisely why they beat the failure mode everyone eventually commits: pausing a rule for a deploy and rediscovering it paused three weeks later, after the incident it would have caught.

Step 5: Grafana OnCall — escalation with teeth

Routing gets the alert to a channel; Grafana OnCall makes sure a person acknowledges it. You point a webhook-type contact point at an OnCall integration URL (exactly like the oncall-critical receiver above), and OnCall takes over with schedules and escalation chains: notify the on-call engineer via mobile push, wait 5 minutes, phone call, wait 10, escalate to the secondary, then the team lead. Schedules sync from Google Calendar or are defined in OnCall directly, and the whole configuration is manageable via the Terraform provider if you want escalation as code too.

One planning note: the self-hosted OnCall OSS project has been in maintenance mode since 2025, with Grafana's investment moving to Grafana Cloud IRM. If you're starting fresh, build on Grafana Cloud IRM rather than deploying OnCall OSS (announcement).

The ceiling: routing is not investigation

Run everything above and you'll have a genuinely good setup — rules in git, a routing tree that respects severity, webhooks handling the mechanical fixes, escalation that can't be slept through. Now look at what still happens at 3 a.m. when a real alert fires.

The pipeline ends at delivery. Grafana's automation moves the alert to the right place with impressive precision. It does not look at the dashboard it came from, query Loki for correlated errors, check whether a deploy landed twenty minutes ago, or compare this firing to the last four. The first fifteen minutes of every incident — the investigation — starts from zero, in a human head, every time.

Webhooks automate the known, not the new. Every webhook responder handles the one failure mode you predicted when you wrote it. Incidents that match no script — most of the interesting ones — fall straight through to a human.

Thresholds decay silently. The 800ms threshold that was right at launch is wrong after the next traffic doubling. Nothing in this stack re-evaluates its own rules; tuning is a recurring engineering tax, and skipping it is how teams drift back to the noise levels part one described.

Context lives in five other tabs. The notification carries labels and a value. The diagnosis lives in Prometheus, Loki, the CI history, and the cloud console — and native Grafana alerting will not go gather it for you.

Native automation, in short, is a world-class dispatcher. It is not an investigator.

What comes next

The layer above dispatch — receiving the firing alert, querying the underlying data sources, correlating with deploys and logs, and proposing or executing the fix under your approval rules — is what CloudThinker agents add on top of Grafana, through a read-only service account connection, with your notification policies and OnCall escalation left fully intact. That's part three of this series.

Or see it against your own Grafana first: Try CloudThinker free — 100 premium credits, no card required.