Dynatrace Automation with Native Tools: The DIY Integration Guide

The most common Dynatrace integration in production today is a problem-notification webhook pointed at Slack. A Davis problem opens, a message lands in #alerts, and the integration project is declared finished. Detection got dramatically better. Response didn't change at all: an engineer still reads the message, opens the problem, runs the same three queries, and decides what to do.

Dynatrace ships far more automation surface than that webhook. This guide is the DIY build-out: notifications with real payloads, the Problems API v2, DQL for triage context, Workflows for standard reactions, and alerting profiles plus maintenance windows for noise control. All native, nothing to buy. In part one we argued the Problems feed, not raw alerts, is the right surface to automate against. Everything below builds on that.

We'll go tool by tool, walk one problem through the finished build, then finish with an honest account of where the ceiling is.

01Push

Notifications

A custom webhook with the full problem object in the payload.

02Read and write

Problems API

List, enrich, comment on and close problems from a script.

03Ask

DQL

The same triage queries against Grail, interactive or scripted.

04React

Workflows

A Davis problem trigger runs the steps you defined in advance.

05Filter

Profiles

Alerting profiles and maintenance windows keep the feed clean.

---

Tool 1: Problem notifications that push full context

Don't poll for problems when Dynatrace will push them. Under Settings → Integration → Problem notifications → Add notification, the Custom integration type gives you a webhook with a templated JSON payload (docs).

The default Slack-style payload is where most teams stop. Send your receiving endpoint something it can act on instead:

{
  "problemId": "{ProblemID}",
  "pid": "{PID}",
  "title": "{ProblemTitle}",
  "state": "{State}",
  "severity": "{ProblemSeverity}",
  "impact": "{ProblemImpact}",
  "tags": "{Tags}",
  "impactedEntities": {ImpactedEntities},
  "url": "{ProblemURL}",
  "details": {ProblemDetailsJSONv2}
}

How to read it. {PID} is the ID you'll use against the Problems API. {State} fires as OPEN and again as RESOLVED, so your receiver must be idempotent and handle both. {ProblemDetailsJSONv2} embeds the full problem object (evidence, impacted entities, root-cause candidate), so a downstream script often doesn't need a second API call at all. Bind the notification to an alerting profile (Tool 5), or you'll forward every problem in the environment, including the ones you deliberately deprioritized.

Tool 2: Problems API v2 to list, enrich, comment and close

For anything the webhook can't carry, the Problems API v2 is the workhorse. Create an access token with the problems.read and problems.write scopes (Access tokens app), and export your environment ID:

export DT_ENV="abc12345"        # your environment ID
export DT_TOKEN="dt0c01.XXXX..."

List everything currently open:

curl -s "https://$DT_ENV.live.dynatrace.com/api/v2/problems?problemSelector=status(\"open\")&pageSize=100" \
  -H "Authorization: Api-Token $DT_TOKEN"

Pull one problem with the fields that matter for triage:

curl -s "https://$DT_ENV.live.dynatrace.com/api/v2/problems/$PROBLEM_ID?fields=evidenceDetails,impactAnalysis,recentComments" \
  -H "Authorization: Api-Token $DT_TOKEN"

How to read it. rootCauseEntity is Davis's root-cause candidate: the thing to verify first, not trust blindly. evidenceDetails lists the events and metric anomalies Davis correlated into this problem. impactAnalysis estimates affected users per impacted entity, which is your tiebreaker when two problems are open at once.

Comment your findings back onto the problem so the timeline lives in Dynatrace, not in a Slack thread:

curl -s -X POST "https://$DT_ENV.live.dynatrace.com/api/v2/problems/$PROBLEM_ID/comments" \
  -H "Authorization: Api-Token $DT_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"message": "Error spike correlates with deploy 2026-07-12.3 on checkout-service.", "context": "triage-bot"}'

And close it with an evidence comment when it's done:

curl -s -X POST "https://$DT_ENV.live.dynatrace.com/api/v2/problems/$PROBLEM_ID/close" \
  -H "Authorization: Api-Token $DT_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"message": "Rolled back to 2026-07-12.2; error rate at baseline for 30 minutes."}'

A problem closed with a one-line cause is worth more than a dashboard of green. It's the dataset your next retro actually uses.

Tool 3: DQL, the triage queries you run every time

Every triage starts with the same questions: what's in the logs, when did it start, does it match a deploy. DQL against Grail answers them in one place, interactively in a Notebook or from a script.

Error logs for the impacted service, bucketed to spot the inflection point:

fetch logs, from: -30m
| filter dt.entity.service == "SERVICE-6C8C6A2E12D2C0A1"
| filter loglevel == "ERROR"
| summarize errors = count(), by: { bin(timestamp, 5m), log.source }
| sort errors desc

Problems themselves are queryable in Grail too, which turns "what's been noisy this week" into a query instead of console archaeology:

fetch dt.davis.problems, from: -7d
| filter event.status == "ACTIVE"
| fields display_id, event.name, event.category, root_cause_entity_name

To run DQL from a script, use the Grail query API with a platform token (scopes storage:logs:read and storage:buckets:read for log queries):

curl -s -X POST "https://$DT_ENV.apps.dynatrace.com/platform/storage/query/v1/query:execute" \
  -H "Authorization: Bearer $DT_PLATFORM_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"query": "fetch logs, from: -30m | filter loglevel == \"ERROR\" | summarize errors = count(), by: { log.source } | sort errors desc"}'

How to read it. The query API is asynchronous by design. A long-running query returns a requestToken instead of results, and you poll for them:

curl -s -G "https://$DT_ENV.apps.dynatrace.com/platform/storage/query/v1/query:poll" \
  --data-urlencode "request-token=$REQUEST_TOKEN" \
  -H "Authorization: Bearer $DT_PLATFORM_TOKEN"

Note the entity ID in the first query, too. Your automation gets it from the webhook payload or the Problems API, which is exactly why Tools 1 to 3 compose.

Tool 4: Workflows for the standard reactions

Dynatrace Workflows (the AutomationEngine) closes the loop for reactions you can define in advance. A workflow starts from a Davis problem trigger, filtered by problem category, tags or affected entity type, and runs a graph of actions: Execute DQL Query, HTTP Request, Run JavaScript, Slack, Jira, ServiceNow.

A triage workflow worth building this week:

1. Trigger: Davis problem trigger, filtered to problems tagged team:payments in the ERROR category.
2. Execute DQL Query: the error-log query from Tool 3, with the affected entity injected from the trigger event.
3. Run JavaScript: shape the query result into a five-line summary.
4. Slack: post the summary to the team channel, in place of a bare problem title.
5. HTTP Request: POST the same summary to /api/v2/problems/{id}/comments so the enrichment is on the problem record.

That one workflow removes the first ten minutes of every incident for that team. The on-call opens Slack and sees what's failing and since when, where it used to say only that something is failing. Build one per problem category, not one grand workflow with twenty branches. Filters are cheap; debugging a mega-workflow isn't.

Tool 5: Alerting profiles and maintenance windows to cut the noise first

Automation amplifies whatever you feed it, so tune the feed. Two native controls do most of it.

Alerting profiles (Settings → Alerting → Problem alerting profiles) decide which problems reach which notification. The underused feature is the delay: only forward a Slowdown problem if it's still open after 15 to 30 minutes, which quietly drops the self-healing blips that make on-call hate the pager. Combine severity rules with tag filters so the payments profile never pages the data team.

Maintenance windows (Settings → General → Maintenance windows, or the Settings API schema builtin:alerting.maintenance-window) suppress alerting, and optionally detection, during planned work. Create them from your deploy pipeline through the Settings API rather than by hand, or the 2 a.m. migration will page exactly the person who scheduled it.

---

One problem through the finished build

Here's what the five tools do together on a typical problem. The details are illustrative.

At 14:06 Davis opens a Failure rate increase problem on checkout-service, tagged team:payments. The alerting profile matches, and because it's an ERROR problem, there's no delay. The workflow fires within seconds: it runs the error-log DQL for the affected service, and the JavaScript step turns the result into a summary: PaymentDeclinedException up from 3 to 410 per 5 minutes since 14:02, all from one log source. Slack gets the summary with the problem link. The same text lands on the problem as a comment.

The on-call reads it at 14:09. That's a real improvement: they skipped the first ten minutes. Now they have to answer the question the workflow can't. Is this a deploy? A change at the payment provider? A feature flag? They open the CI history (a deploy at 14:01), the feature-flag console (nothing), and the provider's status page (green). They roll back, watch the failure rate fall, and close the problem with a comment.

The workflow did the part everyone does the same way every time. The judgment (which of three explanations is true, and whether rollback is safe) took the same twenty minutes it always did.

What each tool removes, and what it leaves

Read down the right-hand column and you get one job description: look at evidence, decide what it means, and act on it safely. Native tooling handles the steps around that job, never the job itself.

The ceiling: what native automation won't do

Run everything above and you'll have a genuinely good setup, better than most. Be clear about its structural limits:

01Coverage

Predicted only

Workflows encode the failure modes you already imagined.

02Horizon

Dynatrace only

IAM edits, capacity events and flag flips sit outside the telemetry.

03Judgment

Still a person

Nothing validates the candidate or picks between rollback and scale-up.

Workflows execute steps you predicted. Every workflow encodes a failure mode you already imagined, with a fixed response. The incident that matters is usually the one you didn't enumerate, and for that one your automation posts context while a person starts from scratch. Coverage grows one hand-built workflow at a time, and each one is code you now maintain.

Investigation stops at Dynatrace's horizon. Davis correlates what Dynatrace observes. The root cause often lives outside that view: an IAM policy change, a cloud provider capacity event, a config flag flipped an hour earlier, a database parameter nobody instrumented. Cross-checking those means a person with five other consoles open.

Judgment is still manual. Nothing here validates the root-cause candidate, chooses between rollback and scale-up, or decides a problem is safe to close. The webhook moved the notification. The person still moves the incident.

What comes next

The gap left over is investigation and action, and that's what an agent layer is for. CloudThinker connects to Dynatrace read-only, subscribes to the Problems feed, enriches each problem with DQL plus the cloud-side context Dynatrace can't see, validates the Davis root-cause candidate, and proposes fixes. Changes go through Auto Mode, so you decide what runs on its own and what waits for approval. That's part three of this series.

If you skipped part one, it explains why Davis problems, not raw metric events, are the surface to build all of this on.

Or see it against your own environment now: Try CloudThinker free with 100 premium credits and no card required.