Sign inBook a Demo

How To

Datadog Automation: Fix Alert Fatigue and Monitor Sprawl First

Most Datadog automation projects fail before they start: 200-monitor estates where dozens of monitors sit muted or in No Data, flappy thresholds nobody trusts, and multi-alerts that turn one incident into forty pages. Part one of our Datadog observability series maps the anatomy of alert fatigue — monitor sprawl, flap, fan-out — plus the over-ingestion costs (custom metric cardinality, log indexing) that prove unmanaged observability grows its own bill. Each pattern comes with a real API command or console path you can run today, and a picture of what good monitor hygiene looks like before you automate anything.

ST
·
datadogobservabilityalertingalertfatiguemonitoringdevops
Cover Image for Datadog Automation: Fix Alert Fatigue and Monitor Sprawl First

Datadog Automation: Fix Alert Fatigue and Monitor Sprawl First

Picture the Datadog account you probably work in: roughly 200 monitors, accumulated over four years and three platform teams. When a page fires at 2 a.m., the on-call engineer opens the same three dashboards — the service overview, the host map, the APM trace view — every single time, regardless of which monitor fired. The monitor told them something is wrong. The next 40 minutes of clicking is how they find out what.

That gap — between a firing monitor and a human doing the actual investigation — is where most Datadog automation efforts should start, and where almost none of them do. Teams jump straight to wiring webhooks into runbooks, or start shopping for a Datadog alternative, when the underlying problem is that their monitor estate produces noise faster than anyone can act on it. Automating noise just delivers noise faster.

This guide maps the problem landscape: the anatomy of alert fatigue on Datadog, which signals actually deserve a page, and the over-ingestion costs that prove nobody is steering the ship. Each pattern comes with a real detection command or console path you can run today.

This is part one of a three-part series. Part two is a hands-on guide to alert automation with Datadog's native tools — composite monitors, downtimes, event correlation, webhooks, Workflow Automation. Part three covers putting an autonomous action layer on top of Datadog alerts with AI agents.

1. Monitor sprawl: nobody owns 200 monitors

What it is. Monitors accumulate the way EBS snapshots do: every incident retro adds two, every new service clones ten from the last service, and nothing ever gets deleted because deleting a monitor feels like deleting a smoke detector. The result is an estate where a large fraction of monitors are muted, permanently in No Data, or watching services that were decommissioned last year.

Why it happens. Creating a monitor takes two minutes and is a visible act of diligence. Deleting one requires understanding why it existed, and the person who created it left the company.

How to detect it. Count your monitors and their states with the Monitors API:

curl -s "https://api.datadoghq.com/api/v1/monitor?page_size=1000&with_downtimes=true" \
  -H "DD-API-KEY: ${DD_API_KEY}" \
  -H "DD-APPLICATION-KEY: ${DD_APP_KEY}" \
  | jq '{total: length,
         no_data: [.[] | select(.overall_state == "No Data")] | length,
         muted:   [.[] | select(((.matching_downtimes // []) | length > 0)
                          or ((.options.silenced // {}) != {}))] | length}'

Or in the console: Monitors → Manage Monitors, then use the left-hand facets to filter by Status: No Data and Muted: true. A monitor that has been muted for 90 days is not a monitor; it's a liability with a name.

Typical impact. In mature accounts it's common to find 20–40% of monitors that are muted, No Data, or have never once transitioned to alert. Every one of them erodes trust in the ones that matter.

2. Flappy monitors: the alerts everyone has learned to ignore

What it is. A monitor with a threshold set too close to normal operating range, an evaluation window too short to smooth transient spikes, or no recovery threshold — so it alerts, recovers, and alerts again a dozen times a day. Within two weeks the team routes it to a Slack channel nobody reads. Within a month, a real incident fires through that same channel and sits unnoticed for an hour.

Why it happens. Thresholds get set once, on launch day, against traffic patterns that no longer exist. Nobody revisits them because the alert "still works" — technically.

How to detect it. Datadog surfaces noisy and flaky monitors itself: check the monitor quality recommendations under Monitors → Quality in the console (docs). For a manual view, open the Events Explorer and query source:alert, group by monitor, and sort by count over the last two weeks — anything alerting more than a few times a day is either a real recurring problem (fix the system) or a flappy monitor (fix the monitor). Recovery thresholds and longer evaluation windows are set per monitor under Advanced options.

Typical impact. A handful of flappy monitors typically generate the majority of total alert volume. Fixing the worst five often cuts pages by half.

3. Multi-alert fan-out: one incident, forty pages

What it is. A multi-alert monitor grouped by {host} or by {pod_name} does exactly what it says: when a shared dependency fails, it pages once per host. One bad database failover becomes 40 simultaneous notifications, each one identical, each one demanding acknowledgment while the on-call engineer is trying to think.

Why it happens. Per-host grouping is the intuitive default when you write the query, and it's correct for genuinely independent failures (one disk filling up). It's catastrophically wrong for correlated failures — and correlated failures are what real incidents are made of.

How to detect it. Find your multi-alert monitors and eyeball the grouping:

curl -s "https://api.datadoghq.com/api/v1/monitor?page_size=1000" \
  -H "DD-API-KEY: ${DD_API_KEY}" \
  -H "DD-APPLICATION-KEY: ${DD_APP_KEY}" \
  | jq -r '.[] | select(.query | contains(" by {")) | "\(.id)\t\(.name)"'

For each, ask: if this fires on 30 hosts at once, do I want 30 pages? If not, the fix is aggregation in the query, a composite monitor, or alert grouping — all covered hands-on in part two.

Typical impact. Fan-out doesn't just annoy; it destroys incident triage. The first ten minutes of a major incident get spent acknowledging notifications instead of reading them.

4. The signals that actually deserve a page

Strip away the sprawl and the flap, and the monitors worth waking someone for are a short list:

  • Symptom-based SLO alerts. Error-rate and latency burn on user-facing endpoints — the thing customers actually feel. Datadog's SLO burn rate alerts page only when the error budget is being consumed fast enough to matter, which makes them naturally resistant to flap.
  • Saturation with a deadline. Disk-will-fill-in-4-hours forecasts, connection pools near exhaustion, queue depth growing without drain. Datadog's forecast and anomaly monitor types exist for exactly this.
  • Dead-man switches. Heartbeat checks on the jobs and pipelines whose silence is the failure mode.

Everything else — CPU on an autoscaled group, a single 5xx, one slow query — is context, not a page. It belongs on the dashboard the investigator opens, or in a ticket, not in PagerDuty at 2 a.m. If your paging list doesn't look like this, that's the first hygiene project, before any datadog webhook automation gets built on top.

5. Over-ingestion, part one: custom metric cardinality

Alert noise has a financial twin: ingestion noise. The same unmanaged growth that produces 200 monitors also produces a Datadog bill that grows faster than your infrastructure, and custom metrics are usually exhibit A.

What it is. Every unique combination of metric name and tag values is a billable custom metric. Tag a metric with user_id or container_id and one line of instrumentation code silently becomes tens of thousands of billable series. Custom metrics beyond your plan's per-host allotment are billed separately (Datadog custom metrics billing).

How to detect it. Pull your top custom metrics by average cardinality:

curl -s "https://api.datadoghq.com/api/v1/usage/top_avg_metrics?month=$(date +%Y-%m)" \
  -H "DD-API-KEY: ${DD_API_KEY}" \
  -H "DD-APPLICATION-KEY: ${DD_APP_KEY}" | jq '.usage[:15]'

Or in the console: Metrics → Summary, click any metric to see its tag cardinality, and check Organization Settings → Plan & Usage → Usage for the account-level trend. Metrics without Limits lets you drop unqueried tag combinations without touching instrumentation code.

Typical impact. It is routine to find that 10 metrics account for the majority of custom metric spend, and that most of their cardinality comes from tags nobody has ever queried.

6. Over-ingestion, part two: indexing every log

What it is. Datadog splits log pricing into cheap ingestion and comparatively expensive indexing — and the default posture is to index everything that arrives. Debug logs from a chatty service, health-check access logs, kube-probe noise: all indexed, all retained, all billed per million events (Datadog pricing).

How to detect it. Graph the estimated usage metric datadog.estimated_usage.logs.ingested_events filtered by the datadog_is_excluded tag (usage metrics docs). If the datadog_is_excluded:true share is near zero, you have no exclusion filters doing any work — meaning nobody has ever decided which logs are worth indexing. Then open Logs → Indexes and look at the top sources by volume.

Typical impact. Teams adding their first serious exclusion filters (health checks, debug level in non-prod, high-volume access logs) commonly cut indexed volume by 30–60% with zero loss of investigative capability — excluded logs can still be archived and rehydrated.

What good monitor hygiene looks like

The healthy end state is boring and specific:

  • Every monitor has an owner tag (team:payments), and unowned monitors get deleted on a schedule, not debated.
  • Paging monitors are symptom-based (SLO burn, saturation-with-deadline, dead-man switches); cause-based monitors notify channels or tickets, never pagers.
  • Multi-alert grouping is deliberate — per-host only where failures are independent; composite monitors or query aggregation where they're not.
  • Recovery thresholds and sensible evaluation windows on everything that has ever flapped.
  • Scheduled downtimes cover every known maintenance window, so mutes are never "temporary and permanent."
  • Ingestion is reviewed with the same cadence as alerts — top custom metrics and top log sources, monthly, with a named owner.

Why the quarterly cleanup doesn't stick

Most teams have done a monitor cleanup. It works — for about six weeks. Then two new services ship, an incident retro adds five monitors, a new SDK version doubles a metric's tag cardinality, and the estate quietly regrows. Hygiene isn't a project; it's a rate problem. Noise is created continuously, so anything less than continuous curation loses.

That's also why the from-dashboard-to-action gap never closes on its own. Even a clean monitor estate still hands the on-call engineer a symptom and wishes them luck — the investigation itself (correlate metrics, logs, traces, recent deploys; form a hypothesis; act) remains manual, at 2 a.m., under pressure.

Where this series goes next

You can close part of the gap with Datadog's own tooling — part two builds it: composite monitors, downtimes, webhooks, and Workflow Automation, with exact configs. The rest of the gap — the investigating and deciding — is what an autonomous action layer is for. CloudThinker agents connect to Datadog read-only, pick up firing monitors, investigate across your metrics, logs, and infrastructure, and propose remediation with your approval — continuously, not quarterly. Try CloudThinker free — 100 premium credits, no card required — or continue with part two.