Datadog Automation with AI Agents: From Firing Monitor to Applied Fix
A monitor fires at 2:47 a.m. The webhook works. The Slack message lands. And then nothing happens, because the alert has been delivered, not investigated. Between "notification posted" and "an engineer has pulled the metrics, read the logs and found the deploy that caused it" sit 20 to 45 minutes of human time. Native Datadog automation can't compress that time, because workflows run the steps you wrote in advance. They don't look at evidence and decide what to check next.
That gap is what this last part of the series is about. Part one took apart alert fatigue on Datadog: monitor sprawl, flapping thresholds, unscoped multi-alerts, and the custom metrics and log indexing that quietly inflate the bill. Part two built everything native features allow: composite monitors, downtimes, event correlation, webhooks and Workflow Automation.
This part covers the layer that's still missing: CloudThinker sits on top of your Datadog account, picks up firing monitors, runs the investigation your on-call would run, and proposes the fix (or, with your explicit permission, applies it). It isn't a Datadog replacement. Monitors, dashboards and pipelines stay where they are. CloudThinker is the action layer on top.
Why workflows stop where investigation starts
Datadog Workflow Automation is good at what it does. Give it a trigger and a sequence, and it will restart a service, open a ticket or post a runbook link every time, reliably. The limit is structural: a workflow is a decision tree you drew before the incident happened.
Real incidents don't follow the tree. The latency monitor on checkout-api fires. Was it the deploy twenty minutes ago, a slow downstream dependency, a noisy neighbor on the node, or a database that started a vacuum? Each answer sends you somewhere different, and the next question depends on what the last one found. An engineer handles that naturally. A workflow can only follow the branches someone predicted.
01Webhook
Delivers
Gets the alert to the right channel. Says nothing about why it fired.
02Workflow
Executes
Runs the steps you wrote in advance. Same steps whatever the evidence says.
03Agent
Investigates
Looks at the evidence, decides what to check next, and writes its reasoning down.
Connecting Datadog: read-only keys, about five minutes
01Observability
Datadog
- Monitors and eventsmonitors_read, events_read
- Metrics and logsmetrics_read, logs_read_data
- Hosts and dashboardshosts_read, dashboards_read
Scoped, read-only app key
02Investigation
CloudThinker
- Pick up firing monitorsWatches monitor state changes
- InvestigateMetrics, logs, APM, recent changes
- Propose the fixWith rollback steps and evidence
No agents installed
03Where the team works
Chat and on-call
Slack or TeamsFindings and approvals in the thread
PagerDutySame escalation path, evidence attached- JiraFollow-ups with the investigation linked
Nothing new to watch
You create an API key and an application key under Organization Settings → API Keys / Application Keys, and scope the application key to read-only authorization scopes: monitors_read, metrics_read, logs_read_data, events_read, dashboards_read, hosts_read. Paste both into CloudThinker, pick your Datadog site (US1, EU1 and so on), and the first scan starts. Datadog's own docs on scoped application keys explain what each scope grants.
No agents to install, no changes to your Datadog configuration, no write access at connection time. The Datadog connection guide has the step-by-step.
What happens when a monitor fires
01Signal
Monitor alerts
A monitor moves to Alert. CloudThinker picks it up within seconds.
02Context
Investigate
Monitor history, related metrics, error logs, APM traces, host state.
03Changes
Correlate
Deploys, config changes and autoscaling events in the same window.
04Output
Finding
What fired, what the evidence shows, likely cause, proposed fix.
1. Pull the monitor context. The definition, its query, the threshold that breached, how often it has fired in 30 days, and whether it's flapping right now. A monitor that fires every day at 02:45 is a different problem from one that has never fired before.
2. Investigate across signals. Related metrics for the affected scope, error logs in the same window, APM traces if the service is instrumented, and host or container state underneath. If p95 latency on checkout-api jumped, the agent checks which endpoints, which downstream calls, and which hosts.
3. Correlate with recent changes. Deploy events, configuration changes, autoscaling activity and anything else in the event stream for the window. Most production incidents trace back to a change, which is why this step comes early.
4. Produce a finding, not a forward. Instead of a raw alert, the channel gets a summary: what fired, what the evidence shows, the most likely cause, and a proposed fix with a rollback note. It lands in the same Slack channel your webhook used to fill with noise.
The difference from the part-two workflow setup is the branching. A workflow follows the steps you wrote. The agent decides what to look at next based on what it just found, and the reasoning is written down so you can check it.
Nothing runs that you haven't allowed
CloudThinker acts within Auto Mode, so you decide what runs on its own and what waits for approval.
Escalation stays intact. If the investigation says "this is a real outage, not noise," it pages exactly the way your current escalation path does, only faster and with the investigation attached. Everything lands in an audit trail: what fired, what was checked, what was proposed, who approved, what changed and when.
What a first scan usually finds
Beyond live alerts, the first connection runs a hygiene and cost pass over the account: the review from parts one and two, done in minutes. The numbers below are illustrative, a composite of what a first scan tends to surface in a mid-market environment with 300 to 600 monitors and a five-figure monthly Datadog bill. Yours will differ.
| Finding | Detail | Impact |
|---|---|---|
| Flapping monitors | 31 monitors alerted 10+ times in 7 days, none actioned | ~60% of total alert volume |
| Monitors with no owner | 84 monitors notify a channel nobody reads or a departed user | Silent blind spots |
| Missing recovery thresholds | 47 monitors flap at the threshold boundary | Repeat pages for one incident |
| Custom metric cardinality growth | 3 metrics added ~180K series via a user_id tag | ~$900/month and climbing |
| Indexed but never queried logs | 2 services index debug logs untouched in 90 days | ~$1,400/month |
| Unmonitored critical services | 4 services with APM traces but no latency or error monitor | Outages found by customers |
| No-data monitors | 19 monitors in No Data for over 30 days | Dead coverage, false confidence |
Look at the shape rather than the numbers. The cost rows are the over-ingestion story from part one showing up as line items, and they often pay for the whole exercise. And most rows aren't "add more monitoring". They're "your current monitoring is misleading you in specific, fixable ways." Each row comes with a proposed fix, gated by approval.
Why this beats the traditional setup
| Webhooks and workflows | With CloudThinker on top | |
|---|---|---|
| At 2:47 a.m. | A message in Slack | A finding with evidence and a likely cause |
| Unfamiliar alerts | Fall outside the drawn branches | Investigated from the evidence |
| Change correlation | Manual, usually last | Automatic, usually first |
| Hygiene and cost | Quarterly cleanup, then regrowth | Continuous, with proposed fixes |
| Actions | Whatever the workflow was told | Gated by Auto Mode, audited |
Prompts to try in your first session
CloudThinker is conversational. You ask in plain language, and the answer cites the Datadog data so you can verify it rather than trust it:
"The p95 latency monitor on checkout-api just fired. Investigate: pull the monitor history, related APM and error logs for the last hour, and check for deploys or config changes in the same window. Propose only, change nothing."
"Review the payments team's monitors: which ones flapped more than five times last week, and what threshold or evaluation-window changes would stop the noise without losing real signal?"
"Which custom metrics grew the most in cardinality this quarter, and which of them appear in no dashboard and no monitor?"
Run the first one during your next real incident. Compare the three-minute investigation with what your on-call assembled by hand, and you'll know quickly whether it earns a place in your escalation path.
What the agents won't do
"AI agent connected to production monitoring" should raise questions. Here are the answers:
- Read-only by default. The connection uses read scopes only. Muting a monitor, editing a threshold or scheduling a downtime needs write scopes, and every write goes through Auto Mode.
- No silent changes. Every write is logged, and anything you haven't let run on its own waits for approval in chat.
- No bypassing escalation. The agent adds to your paging path. It doesn't decide an incident isn't worth waking anyone for. Suppression rules are yours, explicit and logged.
- No unauditable actions. Every investigation, proposal, approval and change is in the audit trail with its evidence.
If you built the part-two automation, keep it. Webhooks and workflows are the right tool for paths that are already deterministic. Put the agent on the part that never was: the investigation and the decision.
Closing the loop
Part one showed why Datadog alert volume outgrows human attention. Part two showed how far native automation goes, and where it stops: workflows execute, they don't investigate. The distance between "alert delivered" and "cause found, fix proposed" is what an agent layer closes. Teams that add it on top of an existing Datadog setup typically cut time to diagnosis from tens of minutes to under five, and recover four figures a month in ingestion waste along the way.
Try CloudThinker free with 100 premium credits and no card required, and follow the Datadog connection guide to see your next firing monitor arrive already investigated.