Sign inBook a Demo

How To

Dynatrace Integration Guide: From Davis Problems to Faster Action

A Dynatrace integration should start where the platform stops: Davis AI detects problems in seconds, but remediation still waits for a human. This guide maps what Dynatrace already solves — Davis problems vs raw alerts, the Problems API v2 as the right integration surface, DQL on Grail as the query layer — plus the signals beyond the problem feed: SLO burn, synthetic failures, and host saturation trends. Includes a copy-pasteable Problems API call and DQL triage query, and the sober math of time-to-detection vs time-to-action. Part one of our three-part Dynatrace observability series.

ST
·
dynatraceobservabilityapmdqlmonitoringincidentresponse
Cover Image for Dynatrace Integration Guide: From Davis Problems to Faster Action

Dynatrace Integration Guide: From Davis Problems to Faster Action

Davis flagged the problem at 02:14. Root-cause candidate identified — a saturated connection pool on one service instance — impact analysis attached, seven downstream services listed, affected user count estimated. Ninety seconds from first symptom to a complete problem card. The fix landed at 06:20, when the on-call engineer woke up, read the card, agreed with everything on it, and restarted the service.

That four-hour gap is what this guide is about. Any Dynatrace integration worth building starts from an honest map of what the platform already solves — which is more than almost any other observability tool — and where the work still stops and waits for a human. Get that map wrong and you'll spend engineering time rebuilding detection Dynatrace already does; get it right and every hour you invest lands on the actual bottleneck: acting on what Davis found.

This is part one of a three-part series. Part two is a hands-on guide to automating Dynatrace responses with native tools — notification webhooks, the Problems API, and AutomationEngine workflows. Part three covers putting an AI agent on the Dynatrace problem feed so investigation and remediation stop waiting for business hours.

What Davis solves that raw alerts never did

Most monitoring stacks emit alerts: a threshold crossed, a check failed, a metric spiked. One failing database typically produces dozens of them — high latency on every calling service, error-rate breaches, queue depth warnings — and the on-call engineer's first job is archaeology: figuring out which twelve alerts are the same incident.

Davis AI works differently. It's causal, not purely statistical: it combines the live topology model (Smartscape), distributed traces, and event correlation to group related symptoms into a single problem. A Davis problem is not an alert. It carries:

  • A root-cause candidate. Not "CPU is high" but "this specific process on this specific host degraded first, and here's the causal chain to everything else."
  • Impact analysis. Which services, applications, and (estimated) real users are affected — which is what actually determines severity.
  • Correlated evidence. The individual events, anomalies, and deployments Davis folded into the problem, with timestamps.

The practical effect: instead of 40 notifications, you get one problem card. Open the Problems app in the Dynatrace UI and you're looking at a queue of incidents, not a firehose of symptoms. If you've lived through alert fatigue on threshold-based tooling, this alone justifies the platform.

Hold on to that framing, because it defines the integration surface. Anything you bolt onto Dynatrace — ticketing, chat, runbooks, automation — should consume problems, not raw metric events. Integrating at the metric-alert level throws away the correlation work Davis already did.

The Problems API: the right place to plug in

The Problems API v2 exposes the same feed the Problems app shows. Everything on the problem card — root-cause entity, impacted entities, evidence, severity, status — is available as JSON. You'll need an API token with the problems.read scope (Settings → Access tokens in classic environments, or via the Account Management UI on the latest platform).

List currently open problems:

curl -s "https://abc12345.live.dynatrace.com/api/v2/problems?problemSelector=status(%22open%22)&pageSize=10" \
  -H "Authorization: Api-Token dt0c01.YOUR_TOKEN" | jq '.problems[] | {problemId, title, severityLevel, impactLevel, rootCauseEntity: .rootCauseEntity.name}'

The fields worth building on: rootCauseEntity (Davis's candidate), impactedEntities and affectedEntities (blast radius), evidenceDetails (the correlated events), and status (problems transition to CLOSED automatically when symptoms clear). That last behavior cuts both ways — auto-close keeps the queue honest, but it also means a recurring problem that self-resolves every night at 3 a.m. can flap for weeks without anyone being forced to look at it. Pull the feed periodically and count reopened titles; chronic flappers are your highest-leverage fixes.

DQL: the question layer on top of Grail

Detection tells you that something broke and where. Investigation still needs specifics: which errors, since when, correlated with what. On the current Dynatrace platform, logs, metrics, events, and traces land in Grail, and you query all of it with Dynatrace Query Language (DQL) — from a Notebook, a dashboard tile, or the API.

The canonical triage query — error logs around a problem's root-cause host, bucketed to spot the onset:

fetch logs, from: -30m
| filter dt.entity.host == "HOST-ABC123DEF456"
| filter loglevel == "ERROR"
| summarize errors = count(), by: { bin(timestamp, 5m), log.source }
| sort errors desc

Davis problems themselves are queryable the same way — they're stored in Grail as events with event.kind == "DAVIS_PROBLEM" — so you can answer meta-questions like "which service generated the most problems this month" in one query instead of exporting the feed.

DQL matters for integration planning because it's the piece that makes enrichment automatable. Every manual investigation is a sequence of DQL queries someone types at 3 a.m. Written down, they're a runbook. Executed automatically when a problem opens, they're context that arrives with the page instead of 40 minutes after it.

Three signals that matter beyond the problem feed

Problems cover anomalies Davis can causally explain. Three signal classes deserve their own attention because they degrade quietly or fire outside the problem model's sweet spot:

Service-level objectives. SLOs in Dynatrace track error-budget burn — the difference between "an incident happened" and "we can no longer afford incidents this month." Check the Service-Level Objectives app for anything trending toward breach. Burn rate is a decision signal: it should change how aggressively you respond to the next problem on that service, which is exactly the kind of judgment no threshold encodes.

Synthetic failures. Synthetic monitors (browser clickpaths and HTTP checks) are your outside-in view — they catch expired certificates, broken DNS, and third-party failures that produce zero anomalies in your own telemetry. Review failure patterns under Synthetic and make sure global outage criteria (how many locations must fail, for how long) match what you'd actually page on; the default may be noisier or laxer than your reality.

Host health trends. Slow burns — a disk filling at a steady 2% a day, memory creeping across weeks — are trajectories, not anomalies. Davis raises predictive disk-full events for some of these, but a periodic saturation review of Hosts (or a scheduled DQL query over dt.host.disk.used.percent) catches the rest before they become 2 a.m. problems.

None of these page anyone by default with useful urgency. They're the signals a weekly human review is supposed to catch — and the review is the first thing skipped in a busy sprint.

Time-to-detection is solved. Time-to-action isn't.

Here is the sober arithmetic. On a well-instrumented Dynatrace environment, detection latency is minutes at worst — often under two. But the incident clock doesn't stop at detection. Someone still has to:

  1. Pick up the problem card — after the page, the wake-up, the context switch.
  2. Validate the root-cause candidate. Davis is good, but it can only reason over what it observes. Was there a config change in your cloud account? A deploy from a pipeline Dynatrace isn't watching? A quota change on the database service? That correlation is manual.
  3. Decide and act. Restart, roll back, scale, failover — under change-management rules, usually via a different console than the one showing the problem.
  4. Verify and close — confirm symptoms cleared, write up what happened.

If detection takes 90 seconds and the rest takes four hours, detection is under 1% of incident duration. Every additional euro spent sharpening detection optimizes the fraction that's already solved. The 99% — the action gap — is organizational: problems wait in queues, queues wait for humans, and humans have working hours, on-call fatigue, and competing pages. Dynatrace hands you the best problem card in the industry, and then the card sits there.

Closing that gap is an engineering project with two routes. You can build it yourself from Dynatrace's native pieces — problem notification webhooks, the Problems API, DQL-powered enrichment, and AutomationEngine workflows — which is exactly what part two of this series walks through, including where the ceiling is. Or you can put an autonomous layer on top of the problem feed, which is part three.

Put an action layer on the problem feed

This is the gap CloudThinker closes. CloudThinker agents connect to Dynatrace read-only, subscribe to the Problems feed, enrich each problem with DQL context plus what Dynatrace can't see — recent config changes, cloud-side state — validate the Davis root-cause candidate, and propose or apply remediation under autonomy levels you control, with escalation intact. Try CloudThinker free — 100 premium credits, no card required — or continue with the native-tools automation guide.