Sign inContact Us

How To

Datadog Automation: Fix Alert Fatigue and Monitor Sprawl First

Most Datadog automation projects fail before they start: 200-monitor estates where dozens of monitors sit muted or in No Data, flappy thresholds nobody trusts, and multi-alerts that turn one incident into forty pages. Part one of our Datadog observability series maps the anatomy of alert fatigue (monitor sprawl, flapping, fan-out) plus the over-ingestion costs (custom metric cardinality, log indexing) that prove unmanaged observability grows its own bill. Each pattern comes with a real API command or console path you can run today, and a picture of what good monitor hygiene looks like before you automate anything.

ST
8 min read
Cover Image for Datadog Automation: Fix Alert Fatigue and Monitor Sprawl First

Datadog Automation: Fix Alert Fatigue and Monitor Sprawl First

Picture the Datadog account you probably work in. Roughly 200 monitors, built up over four years by three platform teams. When a page fires at 2 a.m., the on-call engineer opens the same three dashboards every time (the service overview, the host map, the APM trace view) no matter which monitor fired. The monitor said something is wrong. The next 40 minutes of clicking is how they find out what.

That gap between a firing monitor and someone doing the actual investigation is where Datadog automation should start, and where almost none of it does. Teams jump straight to wiring webhooks into runbooks, or start pricing a Datadog alternative, when the real problem is that the monitor estate produces noise faster than anyone can act on it. Automating noise only delivers it faster.

This guide maps that problem: how alert fatigue builds up on Datadog, which signals deserve a page, and the over-ingestion costs that show nobody is steering. Every pattern comes with a command or console path you can run today.

The six patterns below fall into two groups. The first four are about attention: monitors that cost on-call focus without paying it back. The last two are about money: the same unmanaged growth showing up on the bill.

01Attention

Monitor sprawl

Hundreds of monitors, many muted, in No Data, or watching services that no longer exist.

02Attention

Flapping

Thresholds inside the normal range, so alerts open and close all day and get ignored.

03Attention

Fan-out

Multi-alerts grouped by host turn one shared failure into dozens of pages.

04Attention

Wrong pages

Cause-based alerts paging people for things customers never feel.

05Cost

Metric cardinality

One high-cardinality tag turns a metric into thousands of billable series.

06Cost

Indexing everything

Debug logs and health checks indexed and retained at full price.

The six patterns this guide covers, each with a way to detect it today.

1. Monitor sprawl: nobody owns 200 monitors

What it is. Monitors pile up the way EBS snapshots do. Every incident retro adds two, every new service clones ten from the last one, and nothing gets deleted because deleting a monitor feels like removing a smoke detector. You end up with an estate where a large share of monitors are muted, stuck in No Data, or watching services that were shut down last year.

Why it happens. Creating a monitor takes two minutes and looks diligent. Deleting one means understanding why it existed, and the person who made it has left.

How to detect it. Count your monitors and their states with the Monitors API:

curl -s "https://api.datadoghq.com/api/v1/monitor?page_size=1000&with_downtimes=true" \
  -H "DD-API-KEY: ${DD_API_KEY}" \
  -H "DD-APPLICATION-KEY: ${DD_APP_KEY}" \
  | jq '{total: length,
         no_data: [.[] | select(.overall_state == "No Data")] | length,
         muted:   [.[] | select(((.matching_downtimes // []) | length > 0)
                          or ((.options.silenced // {}) != {}))] | length}'

Or in the console: Monitors → Manage Monitors, then filter the left-hand facets by Status: No Data and Muted: true. A monitor muted for 90 days isn't a monitor. It's a liability with a name.

Typical impact. In mature accounts it's common to find 20 to 40% of monitors muted, in No Data, or never once in alert. Each one wears down trust in the monitors that matter.

2. Flapping monitors: the alerts everyone has learned to ignore

What it is. A monitor whose threshold sits too close to the normal range, whose evaluation window is too short to smooth spikes, or that has no recovery threshold. It alerts, recovers and alerts again a dozen times a day. Within two weeks the team routes it to a channel nobody reads. Within a month a real incident fires into that same channel and sits there for an hour.

Why it happens. Thresholds get set once, on launch day, against traffic that no longer exists. Nobody revisits them because the alert still technically works.

How to detect it. Datadog flags noisy and flaky monitors itself under Monitors → Quality (docs). For a manual view, open the Events Explorer, query source:alert, group by monitor and sort by count over two weeks. Anything alerting more than a few times a day is either a real recurring problem (fix the system) or a flapping monitor (fix the monitor). Recovery thresholds and longer evaluation windows live in each monitor's Advanced options.

Typical impact. A handful of flapping monitors usually generate most of the alert volume. Fixing the worst five often halves the pages.

3. Multi-alert fan-out: one incident, forty pages

What it is. A multi-alert monitor grouped by {host} or by {pod_name} does exactly what it says. When a shared dependency fails, it pages once per host. One bad database failover becomes 40 identical notifications, each demanding an acknowledgment while the on-call is trying to think.

Why it happens. Per-host grouping is the natural default when you write the query, and it's right for independent failures, like one disk filling up. It's wrong for correlated failures, and correlated failures are what real incidents are made of.

How to detect it. List your multi-alert monitors and look at the grouping:

curl -s "https://api.datadoghq.com/api/v1/monitor?page_size=1000" \
  -H "DD-API-KEY: ${DD_API_KEY}" \
  -H "DD-APPLICATION-KEY: ${DD_APP_KEY}" \
  | jq -r '.[] | select(.query | contains(" by {")) | "\(.id)\t\(.name)"'

For each one, ask: if this fires on 30 hosts at once, do I want 30 pages? If not, the fix is aggregation in the query, a composite monitor, or alert grouping, all built hands-on in part two.

Typical impact. Fan-out doesn't just annoy people. It wrecks triage. The first ten minutes of a major incident go to acknowledging notifications instead of reading them.

4. The signals that deserve a page

Take away the sprawl and the flapping, and the monitors worth waking someone for are a short list:

01Symptom

SLO burn rate

Error rate and latency burn on user-facing endpoints. Pages only when the error budget is going fast enough to matter.

02Deadline

Saturation

Disk fills in four hours, a pool near exhaustion, a queue growing without drain. Forecast and anomaly monitors exist for this.

03Silence

Dead-man switch

Heartbeats on the jobs and pipelines whose failure mode is going quiet.

Everything else is context for the investigation, not a reason to wake someone.

Datadog's SLO burn rate alerts are naturally resistant to flapping because they measure budget consumption, not single spikes. CPU on an autoscaled group, a single 5xx, one slow query: those belong on the dashboard the investigator opens, or in a ticket. Not in PagerDuty at 2 a.m. If your paging list doesn't look like this, that's the first hygiene project, before any webhook automation gets built on top.

5. Over-ingestion, part one: custom metric cardinality

Alert noise has a financial twin: ingestion noise. The same unmanaged growth that produced 200 monitors also produces a bill that grows faster than your infrastructure, and custom metrics are usually the first exhibit.

What it is. Every unique combination of metric name and tag values is a billable custom metric. Tag a metric with user_id or container_id and one line of instrumentation quietly becomes tens of thousands of series. Custom metrics beyond your plan's per-host allotment are billed separately (custom metrics billing).

How to detect it. Pull your top custom metrics by average cardinality:

curl -s "https://api.datadoghq.com/api/v1/usage/top_avg_metrics?month=$(date +%Y-%m)" \
  -H "DD-API-KEY: ${DD_API_KEY}" \
  -H "DD-APPLICATION-KEY: ${DD_APP_KEY}" | jq '.usage[:15]'

Or in the console: Metrics → Summary, open any metric to see its tag cardinality, and check Organization Settings → Plan & Usage → Usage for the trend. Metrics without Limits lets you drop tag combinations nobody queries without touching instrumentation.

Typical impact. It's routine to find that 10 metrics make up most of the custom metric spend, and that most of their cardinality comes from tags nobody has ever queried.

6. Over-ingestion, part two: indexing every log

What it is. Datadog prices log ingestion cheaply and indexing much more expensively, and the default is to index everything that arrives. Debug logs from a chatty service, health-check access logs, kube-probe noise: all indexed, retained and billed per million events (pricing).

How to detect it. Graph datadog.estimated_usage.logs.ingested_events split by the datadog_is_excluded tag (usage metrics). If the datadog_is_excluded:true share is near zero, no exclusion filter is doing any work, which means nobody has ever decided which logs are worth indexing. Then open Logs → Indexes and look at the top sources by volume.

Typical impact. Teams adding their first real exclusion filters (health checks, debug level outside production, high-volume access logs) often cut indexed volume by 30 to 60% with no loss of investigative ability. Excluded logs can still be archived and rehydrated.

What good monitor hygiene looks like

The healthy end state is boring and specific:

  • Every monitor has an owner tag (team:payments), and unowned monitors get deleted on a schedule, not debated.
  • Paging monitors are symptom-based. SLO burn, saturation with a deadline, dead-man switches. Cause-based monitors go to channels or tickets, never pagers.
  • Multi-alert grouping is deliberate. Per host only where failures are independent, composite monitors or aggregation where they aren't.
  • Recovery thresholds and sensible evaluation windows on everything that has ever flapped.
  • Scheduled downtimes cover known maintenance, so no mute is "temporary" forever.
  • Ingestion is reviewed on the same cadence as alerts. Top custom metrics and top log sources, monthly, with a named owner.

Why the quarterly cleanup doesn't stick

Most teams have done a monitor cleanup. It works for about six weeks. Then two services ship, a retro adds five monitors, an SDK upgrade doubles a metric's cardinality, and the estate quietly grows back.

01Week 0

Cleanup

A focused sprint deletes, retunes and re-owns monitors.

02Weeks 1 to 6

Regrowth

New services, retros and SDK changes add noise every week.

03Quarter end

Back to noisy

Pages climb, trust drops, the next cleanup gets scheduled.

Noise is created continuously. Anything less than continuous curation loses.

Hygiene isn't a project. It's a rate problem. And even a clean estate still hands the on-call a symptom and wishes them luck: correlating metrics, logs, traces and recent deploys, forming a hypothesis and acting all stay manual, at 2 a.m., under pressure.

Where this series goes next

You can close part of that gap with Datadog's own tooling. Part two builds it with exact configs: composite monitors, downtimes, webhooks and Workflow Automation. The rest of the gap, the investigating and deciding, is what an autonomous action layer is for, and part three shows how CloudThinker connects to Datadog read-only, picks up firing monitors, investigates across metrics, logs and infrastructure, and proposes fixes for your approval.

Try CloudThinker free with 100 premium credits and no card required, or continue with part two.

datadogobservabilityalertingalertfatiguemonitoringdevops