Dynatrace Integration with AI Agents: Automating Problem Remediation

A Failure rate increase problem opens on billing-api at 14:04. Davis has already done its part: root-cause candidate (connection pool saturation on the billing-api process group), impact analysis, four affected services. The page goes out. The on-call acknowledges at 14:19, reads the card, opens a Notebook, runs two DQL queries, checks the deploy pipeline in another tab, finds a merge that shipped at 14:02, and asks in Slack whether rolling back is safe. The fix lands at 14:51.

Dynatrace solved detection years ago, and Davis got diagnosis most of the way there. What stays manual is the last mile: someone reads the problem, confirms the candidate, checks what changed outside Dynatrace's view, and turns it into a fix. Your time to detect is measured in seconds. Your time to remediate is measured in whoever is on call.

This final part of the series is about closing that gap with a Dynatrace integration that treats Davis problems, not raw alerts, as the unit of work. Part one covered why the Problems feed and DQL are the right surfaces to build on. Part two pushed the native tools to their ceiling: problem notifications, the Problems API v2, and AutomationEngine workflows that run predefined steps well but investigate nothing.

Here we add the layer that investigates. CloudThinker subscribes to your Problems feed, enriches each problem with DQL context and with the things Dynatrace can't see, validates the Davis root-cause candidate, and proposes a fix with an evidence trail (or applies it, within the limits you set). It isn't a Dynatrace replacement. OneAgent keeps collecting, Davis keeps diagnosing. CloudThinker is the action layer on top.

---

Why workflows stop where validation starts

Part two's payments workflow runs the error-log query and posts a summary. That saves the first ten minutes. It can't do the next thing, because the next thing depends on what the summary says. If the errors began right after a deploy, check the diff and the rollback path. If they didn't, check the database entity, then the cloud account, then the quota dashboard. A workflow is a graph you drew before the incident. Validating a root-cause candidate is a sequence of questions where each one depends on the last answer, and some of the answers live outside Dynatrace entirely.

That's the structural limit. It's why problems with a perfectly good candidate still wait half an hour for a person.

---

Connection: one scoped token, read-only, about five minutes

01Observability

Dynatrace

• Davis problemsproblems.read
• Topology and eventsentities.read, events.read, metrics.read
• Logs in Grailstorage:logs:read for DQL

Scoped, read-only token

02Investigation

CloudThinker

• Subscribe to problemsPicks up each new Davis problem
• Validate the candidateDQL plus cloud and repo changes
• Propose and closeFix with rollback, evidence comment

No agents installed

03Where the team works

Chat and on-call

• Slack or TeamsFindings and approvals in the thread
• PagerDutySame escalation path, evidence attached
• JiraFollow-ups with the investigation linked

Nothing new to watch

CloudThinker connects to Dynatrace with an access token you create yourself, so the permission boundary is visible before anything runs. The read-only baseline:

• problems.read: the Problems feed and per-problem detail, including the Davis root-cause candidate
• entities.read: the Smartscape topology behind each affected entity
• metrics.read, events.read: metric context and the event stream (deployments, config change events)
• Grail query scopes (storage:logs:read, storage:events:read, storage:buckets:read) if you want DQL run against logs and events

That's the whole grant at connection time. problems.write, needed later to post comments and close problems, is a separate, deliberate step, not part of the default. You generate the token, paste it with your environment URL into CloudThinker, and the first pass over your open problems starts. Budget about five minutes. The Dynatrace connection guide lists the exact scopes, so your security review reads the same list you do.

---

What happens when a problem opens

CloudThinker watches the Problems feed, the same GET /api/v2/problems surface you polled by hand in part two. When a new problem opens, the sequence looks like what a good on-call engineer does, compressed from half an hour into two or three minutes:

01Signal

Problem opens

The full record: candidate, impact, affected entities.

02Enrich

DQL

Error logs and event bursts for the problem window.

03Outside

Changes

Deploys, IAM edits, scaling and quotas Dynatrace never saw.

04Validate

Confirm or challenge

Independent evidence for or against the Davis candidate.

05Output

Proposed fix

Specific remediation with rollback, waiting for approval.

1. Take the problem, not the symptoms. Pull the full problem record: affected entities, impact analysis and the Davis root-cause candidate. Building on problems instead of raw metric alerts keeps the deduplication and causal correlation Davis already did, instead of redoing it worse.

2. Enrich with DQL. For the affected entities and the problem window, query Grail for the evidence a person would open next: error logs, event bursts, related failure rates.

3. Add what Dynatrace can't see. No native workflow can do this step, because it leaves Dynatrace entirely. With your cloud accounts and repositories also connected, CloudThinker checks the surrounding state: the merge that deployed twelve minutes before the problem opened, the security group edit, the autoscaling event, the database parameter change, the quota that quietly hit its ceiling. Davis correlates brilliantly within its own telemetry. Most real root causes involve at least one fact outside it.

4. Validate the Davis candidate. The goal isn't to out-diagnose Davis. It's to confirm or challenge the candidate with independent evidence. When the evidence disagrees, the finding says so and shows both sides.

5. Propose the fix. The output is a finding in your chat channel: what opened, what the evidence shows, whether the candidate held up, and a specific remediation with rollback notes, ready for approval when it needs one.

---

The 14:04 problem, investigated

Back to the opening problem. The details are illustrative.

Davis says the billing-api connection pool is saturated. That's a symptom with at least three possible causes, and each predicts something different. If traffic outgrew the pool, request volume is above its normal peak. If the 14:02 merge leaks connections, active connections climb steadily on the new version and never return. If the database slowed down, query time rises and each request holds its connection longer.

The first query checks the shape and onset of the errors:

fetch logs, from: -1h
| filter dt.entity.service == "SERVICE-9A3F2C7D41B8E650"
| filter loglevel == "ERROR"
| summarize errors = count(), by: { bin(timestamp, 5m) }
| sort timestamp desc

Errors start at 14:03, one minute after the merge deployed. Request volume is flat against the same hour last week, which retires the traffic reading. Database response time on the billing-db entity is flat too, which retires the slow-database reading. The diff in the repository shows the merge raised request concurrency for a batch endpoint while the pool stayed at 20.

The finding posted to the channel: Davis's candidate holds, and the cause is the 14:02 merge, which sends more concurrent requests than the pool can serve. Two fixes are proposed with trade-offs. Raising the pool max from 20 to 50 is immediate and reversible, and the database has headroom for it. Rolling back the merge is safer but delays the batch feature. The on-call approves the pool change in Slack at 14:08.

Same diagnosis the on-call reached at 14:51 in the opening. Forty-three minutes earlier.

---

Your escalation path stays yours

CloudThinker acts within Auto Mode, so you decide what runs on its own and what waits for approval. The part that matters for a Dynatrace shop: your escalation path is untouched. If the investigation concludes this is a genuine customer-facing outage, the page goes out exactly as it does today, with the investigation attached instead of a bare problem link. The agent never decides an incident isn't worth waking someone for.

---

Closing the problem with evidence, not silence

Part two showed the mechanics: POST /api/v2/problems/{problemId}/close accepts a closing message. Native workflows can call it, but they can't write anything worth reading. Once a fix is applied and verified, CloudThinker closes the problem with a comment that doubles as the incident record:

curl -X POST \
  "https://{env-id}.live.dynatrace.com/api/v2/problems/{problemId}/close" \
  -H "Authorization: Api-Token $DT_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"message": "Root cause confirmed: connection pool exhaustion in billing-api after 14:02 deploy. Pool max raised 20 to 50 (change approved by j.nguyen). Error rate returned to baseline within 4 minutes. Evidence and timeline in CloudThinker audit log."}'

Six months later, when the same problem pattern reappears, the history is in Dynatrace where the next engineer will actually find it, not in a Slack thread that scrolled away. Every investigation, proposal, approval and executed change also lands in CloudThinker's own audit trail with its evidence.

---

What the first week typically surfaces

Beyond live problem response, the first connection reviews your problem history. Numbers are illustrative: a composite mid-market environment with a few hundred hosts and a few dozen services. Yours will differ.

Look at the shape: only one row is a new detection. The rest are your existing Dynatrace signal being ignored, delayed or drowned in predictable noise, which is the argument that the bottleneck was never detection.

---

Why this beats the traditional setup

The deeper difference is what the on-call starts from. With workflows, they start from context and do the validation. With an agent layer, they start from a validated finding and decide whether to approve it.

---

Prompts worth trying first

CloudThinker is conversational. You ask in plain language, and every answer cites the underlying Dynatrace data so you can verify it rather than trust it:

"A response-time degradation problem just opened on checkout-service. Pull the Davis root-cause candidate, run a DQL error-log query for the affected window, check for deploys or cloud-side changes in the last two hours, and post a summary. Propose only, change nothing."

"Which problems recurred more than five times in the last 30 days without anyone acting on them? For each, tell me whether the right fix is a threshold change, a maintenance window, or an actual defect."

"Sample five problems Davis closed automatically last week and verify the root-cause candidate against logs and events. Flag any where the evidence disagrees."

Run the first one during your next real problem and compare it with what your on-call assembles by hand. That comparison, not this article, is the evaluation.

---

The boundaries, stated plainly

An agent with a token to your production observability platform should raise questions. Here are the answers:

• Read-only by default. The baseline token can't modify anything in Dynatrace or your infrastructure. Acting needs write scopes you grant explicitly, and every change goes through Auto Mode.
• No unchecked changes. Every write follows the Auto Mode rules you set, and every action lands in the audit trail with its evidence.
• No swallowed escalations. Paging behavior is yours. The agent adds evidence to it and never subtracts urgency from it.
• No unauditable actions. Everything is logged with evidence, approver and timestamp, in CloudThinker and, through the closing comment, in Dynatrace itself.

If you built the part-two workflows, keep them. They're the right tool for reactions that are already deterministic. Put the agent on the part that never was: validating the candidate and deciding what to do.

Detection was never the bottleneck

Across this series the pattern held. Dynatrace detects in seconds, Davis diagnoses in minutes, and remediation waits in a queue measured in tens of minutes to days. Native notifications and workflows automate the predictable slice. An agent layer takes the rest: the investigation, the cross-system correlation, the approval-gated fix and the evidence-bearing close. Teams that put an action layer on an existing Dynatrace setup typically cut acknowledgment-to-remediation from the better part of an hour to minutes, and problem noise drops as the recurring offenders finally get fixed instead of re-detected.

Try CloudThinker free with 100 premium credits and no card required, and follow the Dynatrace connection guide to see your own problem history reviewed within the hour.