Datadog Alert Automation with Native Tools (Monitors to Workflows)

The composite monitor on checkout-api fires at 2:47 a.m.: p95 latency and error rate, both over threshold. The webhook posts to #ops-alerts. The workflow you built last quarter wakes up, checks whether anything deployed in the last 30 minutes, finds nothing, and does what its last branch says: page the on-call. Everything worked exactly as configured. The engineer still opens three dashboards and starts from zero.

That's a fair picture of how far Datadog's own automation goes, and it's further than most teams take it. Composite monitors, scheduled downtimes, event correlation, webhooks and Workflow Automation can turn a noisy monitor estate into a tidy pipeline that pages less and does the repetitive fixes by itself. The catch is where the pipeline ends. Somewhere around the fourth branching workflow you notice you're no longer configuring alerts. You're maintaining a runbook engine.

Part one of this series took apart the problem: monitor sprawl, flapping thresholds, multi-alert fan-out, and the over-ingestion costs that ride along. This part is the hands-on build with Datadog's native features only, tool by tool, with configs you can paste. Part three covers the layer that sits on top.

Build the stack in this order. Automation attached to a noisy trigger just delivers noise faster, so the trigger layer comes first and workflows come last.

01Trigger

Monitors

Recovery thresholds, full windows and composites so the trigger means something.

02Silence

Downtimes

Scheduled, scoped by tag, with a message that says why.

03Group

Correlation

Related events folded into one case before anything pages.

04Deliver

Webhooks

Any monitor calls any endpoint, with the alert context in the payload.

05Execute

Workflows

Prebuilt actions, branches and approval steps, run on every trigger.

---

Tool 1: Monitors that don't cry wolf

Fix the trigger before you wire any action to it (monitor types).

Stop flapping with recovery thresholds and full windows

A monitor that alerts at 85% CPU and recovers at 84.9% will flap all day. Give it a recovery threshold well below the alert threshold and a window long enough to smooth spikes. Through the Monitors API:

curl -X POST "https://api.datadoghq.com/api/v1/monitor" \
  -H "DD-API-KEY: ${DD_API_KEY}" \
  -H "DD-APPLICATION-KEY: ${DD_APP_KEY}" \
  -H "Content-Type: application/json" \
  -d '{
    "type": "metric alert",
    "name": "High CPU on {{host.name}} ({{service.name}})",
    "query": "avg(last_15m):avg:system.cpu.user{service:checkout} by {host} > 85",
    "message": "{{#is_alert}}CPU sustained above 85% for 15m. @webhook-remediate-cpu{{/is_alert}}\n{{#is_recovery}}Recovered.{{/is_recovery}} @slack-ops-alerts",
    "tags": ["team:platform", "service:checkout"],
    "options": {
      "thresholds": { "critical": 85, "warning": 75, "critical_recovery": 70 },
      "require_full_window": true,
      "renotify_interval": 60,
      "notify_no_data": false
    }
  }'

Three options do the anti-fatigue work. critical_recovery means the value has to fall to 70 before the monitor can recover and alert again. require_full_window stops it judging partial data. renotify_interval re-notifies every 60 minutes while the problem is still open, instead of never or constantly.

Composite monitors: alert on the intersection

High latency alone might be a deploy settling. A higher error rate alone might be one bad client. Both at once is an incident. Composite monitors combine existing monitors by ID with boolean logic:

12345678 && 23456789        # latency AND error rate both alerting
12345678 && !34567890       # alert only if the maintenance-mode monitor is NOT alerting

Create them under Monitors → New Monitor → Composite, pick the component monitors and set the trigger expression. Attach notifications and webhooks to the composite, and move the components to low-urgency channels. That one change removes a large slice of the "two pages for one problem" pattern from part one.

---

Tool 2: Downtimes, silence you schedule instead of improvise

Deploys, nightly batch jobs and staging environments that sleep overnight all produce predictable alerts. Muting monitors by hand and forgetting to unmute them is the wrong fix. Downtimes are the right one: scoped by tag, scheduled with an RRULE, and visible to everyone.

curl -X POST "https://api.datadoghq.com/api/v2/downtime" \
  -H "DD-API-KEY: ${DD_API_KEY}" \
  -H "DD-APPLICATION-KEY: ${DD_APP_KEY}" \
  -H "Content-Type: application/json" \
  -d '{
    "data": {
      "type": "downtime",
      "attributes": {
        "monitor_identifier": { "monitor_tags": ["env:staging"] },
        "scope": "env:staging",
        "display_timezone": "Australia/Sydney",
        "message": "Staging sleeps outside business hours.",
        "schedule": {
          "timezone": "Australia/Sydney",
          "recurrences": [
            { "start": "2026-07-13T19:00:00", "duration": "12h", "rrule": "FREQ=DAILY;INTERVAL=1" }
          ]
        }
      }
    }
  }'

Enforce two habits. Every downtime gets a message that says why it exists. Someone reviews Monitors → Downtimes monthly, because a stale downtime is how a real incident goes unseen. If your deploy pipeline is scriptable, have it create a one-off downtime scoped to the deploying service and delete it once health checks pass. That alone removes most deploy-window noise.

---

Tool 3: Grouping and correlation, fewer and better events

Two native mechanisms shrink event volume before anything pages.

Multi-alert grouping. A monitor grouped by {host} across 200 hosts is 200 possible pages. If the failure mode is cluster-wide, group by a coarser dimension (by {cluster} or by {service}), or add a coarse monitor that pages and route the per-host one to a ticket queue.

Event correlation. Datadog Event Management can fold related alert events (same service, same window, shared tags) into one case, so a cascading failure arrives as one correlated issue instead of fifteen notifications. Configure it under Service Management → Event Management → Correlation, keyed on tags like service, env and cluster. Turn it on before you build anything downstream, because every automation you attach runs once per event it receives.

---

Tool 4: Webhooks, the escape hatch to your own code

The Webhooks integration lets any monitor call any HTTP endpoint. Set it up under Integrations → Webhooks, name it (say remediate-cpu), point it at your endpoint and define the payload with Datadog's template variables:

{
  "monitor_id": "$ALERT_ID",
  "title": "$EVENT_TITLE",
  "status": "$ALERT_STATUS",
  "transition": "$ALERT_TRANSITION",
  "scope": "$ALERT_SCOPE",
  "query": "$ALERT_QUERY",
  "hostname": "$HOSTNAME",
  "priority": "$PRIORITY",
  "tags": "$TAGS",
  "event_url": "$LINK"
}

Reference it in any monitor message with @webhook-remediate-cpu. On the receiving side, use $ALERT_TRANSITION to tell Triggered from Recovered, and $ALERT_SCOPE to know which host or service tripped a multi-alert.

What teams build behind that endpoint: a Lambda that restarts a systemd unit, a small service that bumps an ECS desired count, a script that drains a queue backlog. It works, and it brings everything self-hosted automation implies: authentication on the endpoint, retries and idempotency (Datadog sends again on every renotify), secrets to rotate, and monitoring for the remediation itself. Your webhook handler is now production code with its own on-call.

---

Tool 5: Workflow Automation, runbooks as flowcharts

Workflow Automation is Datadog's answer to "stop writing webhook handlers." You build workflows in a visual editor from hundreds of prebuilt actions (AWS, Kubernetes, PagerDuty, Slack, GitHub, Jira), with branches, loops and human approval steps. It's billed per workflow execution on top of your Datadog subscription (pricing).

A representative on-alert workflow for a memory leak in an ECS service:

1. Trigger. The monitor Memory high on {{service.name}} fires and mentions the workflow in its message: @workflow-Restart-ECS-Task. You can pass parameters, for example @workflow-Restart-ECS-Task(service={{service.name}}).
2. Fetch context. An AWS action lists the service's tasks and reads the newest deployment time.
3. Branch. If a deploy happened in the last 30 minutes, skip remediation and page the deploying team with the deploy link.
4. Approval gate. A Slack approval step asks the on-call to approve the restart, or the step is skipped for pre-approved low-risk services.
5. Act. An ECS update-service action forces a new deployment, and a follow-up step polls the monitor until it recovers.
6. Record. Post the outcome to the incident channel and add a note to the monitor event.

This is good tooling. For well-understood, repetitive failures you should use it. Build workflows for your top three "we always do the same thing" alerts and you'll feel the difference within a week.

---

One alert through the native stack

Go back to the 2:47 a.m. composite on checkout-api. Before anyone acts, there are three reasonable readings of it, and each predicts something different in the data.

01Reading 1

Bad deploy

Errors step up at the deploy timestamp, on the new version tag only. The workflow checks this one.

02Reading 2

Slow dependency

Latency lives in downstream spans and timeout errors, across every version.

03Reading 3

Host contention

One or two hosts carry the regression; the rest of the fleet looks normal.

Each reading has a query that confirms or kills it.

A bad deploy shows as an error step at the deploy time, carried by one version tag:

sum:trace.http.request.errors{service:checkout-api,env:prod} by {version}.as_count()

If errors split cleanly by version and the new one owns them, you have your answer. The workflow's step 3 checks a proxy for this (any deploy in 30 minutes). It doesn't check whether the new version is the one failing.

A slow dependency shows as latency in specific resources and timeout errors in logs, spread across every version:

avg:trace.http.request.duration{service:checkout-api,env:prod} by {resource_name}

Then in Logs Explorer, service:checkout-api status:error grouped by error type. A wall of TimeoutError against the payments client points away from your own code.

Host contention shows as the regression concentrated on a few hosts:

avg:system.cpu.user{service:checkout-api,env:prod} by {host}

Two hosts at 95% while the rest sit at 40% is a placement or noisy-neighbor problem, not an application one.

Here's the honest result. The workflow encodes reading one, and only as "was there a deploy." When the answer is no, it pages a human, who then runs the other queries by hand. On this night the answer was reading two, which is the common case: most incidents that aren't deploys are dependencies, and no branch you drew in advance knew which dependency.

---

What each native tool solves, and what it leaves

The pattern down the right-hand column is the same: native automation handles the part you already understood well enough to write down. Anything that needs reading evidence and choosing the next check stays with the on-call.

---

The ceiling: workflows execute, they don't investigate

Before you conclude you need a Datadog alternative or a bigger automation budget, look at what everything above has in common. You had to know the failure and the fix in advance.

01Branches

Drawn in advance

The alert that matches no branch falls through to a human, and it is usually the one that matters.

02Evidence

Nobody reads it

No native step reads the logs, compares traces and forms a hypothesis.

03Upkeep

The library is the job

Every new service, renamed tag or changed runbook is a workflow edit.

04Coverage

Follows effort

You automate the alerts you have seen often. The long tail stays manual.

A workflow is a flowchart. It handles the branches you drew, and the alert that doesn't match any branch (disproportionately the one that matters) falls through to a human at 3 a.m., same as before. When the composite fires, no native feature reads the logs, compares the traces, checks what deployed and forms a hypothesis. It runs step 4 because step 3 returned true.

Upkeep grows with the estate too. Every new service, changed runbook and renamed tag is a workflow edit, and teams that go deep on this path end up with an unofficial "automation owner" whose sprint is maintaining flowcharts. Coverage ends up following effort rather than risk: you automate the alerts you've seen often enough to script, and the long tail, which is most of your alert volume per part one, stays manual.

The approval question you'll meet again

Workflow Automation already forces one decision you'll see again in part three: which actions run unattended and which wait for a person. The Slack approval step in Tool 5 is that line. In part three, CloudThinker acts within Auto Mode, so you make the same call: what runs on its own and what waits for approval. If you've already decided which restarts are safe to run unattended, you've done half the work.

---

What comes next

Native Datadog automation turns known, repeated toil into config. What it can't turn into config is the work between "monitor fired" and "we know what to do": investigate, correlate, decide. That still runs on engineer attention.

Part three covers that layer. CloudThinker connects to Datadog with a read-only application key, picks up firing monitors, runs the three checks above (and whatever the evidence says to check next), and proposes the fix with its reasoning written down. Keep the workflows you built here; they're the right tool for paths that are already deterministic.

If you want to see it against your own monitors first, ask something like this in your first session:

"The composite monitor on checkout-api fired last night. Check the error split by version, latency by resource, and CPU by host for that window, and tell me which reading the evidence supports. Propose only, change nothing."

Try CloudThinker free with 100 premium credits and no card required, or continue with part three.