Dynatrace Integration Guide: From Davis Problems to Faster Action

Davis flagged the problem at 02:14. Root-cause candidate identified (a saturated connection pool on one service instance), impact analysis attached, seven downstream services listed, affected user count estimated. Ninety seconds from first symptom to a complete problem card. The fix landed at 06:20, when the on-call engineer woke up, read the card, agreed with everything on it, and restarted the service.

That four-hour gap is what this guide is about. Any Dynatrace integration worth building starts from an honest map of what the platform already solves (more than almost any other observability tool) and where the work still stops and waits for a person. Get that map wrong and you'll spend engineering time rebuilding detection Dynatrace already does. Get it right and every hour you invest lands on the actual bottleneck: acting on what Davis found.

This is part one of a three-part series. Part two is a hands-on guide to automating Dynatrace responses with native tools: notification webhooks, the Problems API and AutomationEngine workflows. Part three covers putting an AI agent on the Dynatrace problem feed so investigation and remediation stop waiting for business hours.

What Davis solves that raw alerts never did

Most monitoring stacks emit alerts: a threshold crossed, a check failed, a metric spiked. One failing database typically produces dozens of them (high latency on every calling service, error-rate breaches, queue depth warnings), and the on-call engineer's first job is archaeology: working out which twelve alerts are the same incident.

Davis AI works differently. It's causal, not purely statistical. It combines the live topology model (Smartscape), distributed traces and event correlation to group related symptoms into a single problem. A Davis problem isn't an alert. It carries three things:

01Cause

Root-cause candidate

The specific process or host that degraded first, with the causal chain to everything else.

02Impact

Blast radius

Affected services, applications and estimated real users, which is what sets severity.

03Evidence

Correlated events

The anomalies and deployments Davis folded into the problem, with timestamps.

A Davis problem is one card per incident, not one alert per symptom.

The practical effect: instead of 40 notifications, you get one problem card. Open the Problems app and you're looking at a queue of incidents, not a firehose of symptoms. If you've lived through alert fatigue on threshold-based tooling, that alone justifies the platform.

Hold on to that framing, because it defines the integration surface. Anything you connect to Dynatrace (ticketing, chat, runbooks, automation) should consume problems, not raw metric events. Integrating at the metric-alert level throws away the correlation work Davis already did.

Why integrating on raw events breaks

Teams coming from threshold tooling often wire Dynatrace the way they wired the old stack: a metric event per condition, each routed to Slack or a ticketing queue. It breaks for structural reasons:

Every symptom becomes its own ticket. The database failure that Davis folds into one problem arrives as forty events, and the deduplication moves back to a person.
The causal chain is lost. A metric event knows its own threshold. It doesn't know which entity degraded first or what depends on it.
Severity has to be guessed. Impact analysis (which services and how many users) lives on the problem, not on the event.
Thresholds need tuning forever. Davis baselines adapt to the workload. Static metric events don't, so the tuning debt comes back.

	Integrating on raw metric events	Integrating on Davis problems
One failing database	Dozens of tickets	One problem with a root-cause candidate
First question	"Are these the same incident?"	"Is the candidate right?"
Severity	Guessed from the metric name	Read from impact analysis
Maintenance	Per-threshold tuning	Baselines adapt on their own

The deeper difference is where triage starts. Raw events start the on-call at grouping. Problems start them at validation, which is ten or twenty minutes further along.

The Problems API: the right place to plug in

The Problems API v2 exposes the same feed the Problems app shows. Everything on the problem card (root-cause entity, impacted entities, evidence, severity, status) is available as JSON. You'll need an API token with the problems.read scope (Settings → Access tokens in classic environments, or the Account Management UI on the latest platform).

List currently open problems:

curl -s "https://abc12345.live.dynatrace.com/api/v2/problems?problemSelector=status(%22open%22)&pageSize=10" \
  -H "Authorization: Api-Token dt0c01.YOUR_TOKEN" | jq '.problems[] | {problemId, title, severityLevel, impactLevel, rootCauseEntity: .rootCauseEntity.name}'

The fields worth building on: rootCauseEntity (Davis's candidate), impactedEntities and affectedEntities (blast radius), evidenceDetails (the correlated events) and status. Problems move to CLOSED automatically when symptoms clear, and that cuts both ways. Auto-close keeps the queue honest, but it also means a recurring problem that self-resolves every night at 3 a.m. can flap for weeks without anyone being forced to look at it. Pull the feed periodically and count repeated titles. Chronic flappers are your highest-leverage fixes.

DQL: the question layer on top of Grail

Detection tells you that something broke and where. Investigation still needs specifics: which errors, since when, correlated with what. On the current Dynatrace platform, logs, metrics, events and traces land in Grail, and you query all of it with Dynatrace Query Language (DQL) from a Notebook, a dashboard tile or the API.

Davis problems themselves are queryable the same way. They're stored in Grail as events with event.kind == "DAVIS_PROBLEM", so a meta-question like "which entity generated the most problems this month" is one query instead of an export:

fetch events, from: -30d
| filter event.kind == "DAVIS_PROBLEM"
| summarize problems = countDistinct(display_id), by: { root_cause_entity_name }
| sort problems desc
| limit 10

DQL matters for integration planning because it's the piece that makes enrichment automatable. Every manual investigation is a sequence of DQL queries someone types at 3 a.m. Written down, they're a runbook. Run automatically when a problem opens, they're context that arrives with the page instead of 40 minutes after it.

Validating the 02:14 candidate, hypothesis first

Davis said: connection pool saturated on one instance of billing-api. A good engineer doesn't restart on that alone. They ask what could have saturated the pool, because each answer needs a different fix:

Traffic outgrew the pool. Predicts request volume above its usual peak, spread across all instances.
A deploy leaks connections. Predicts a deployment event shortly before onset, and errors only on instances running the new version.
The database got slow. Predicts rising query time on the database entity, so each request holds its connection longer.
Something outside Dynatrace changed. Predicts no signal in the telemetry at all, and a change in the cloud account or a parameter group.

The first query checks the onset and shape of the errors on the root-cause host:

fetch logs, from: -30m
| filter dt.entity.host == "HOST-ABC123DEF456"
| filter loglevel == "ERROR"
| summarize errors = count(), by: { bin(timestamp, 5m), log.source }
| sort errors desc

Suppose it shows pool exhausted errors starting at 02:09, all from one log source, and the other instances clean. That rules out a traffic spike (it would hit every instance). The second query looks for a change in the window:

fetch events, from: -2h
| filter event.type == "CUSTOM_DEPLOYMENT" or event.type == "CUSTOM_CONFIGURATION"
| fields timestamp, event.type, event.name
| sort timestamp asc

A deployment event on that one instance at 02:05 makes the leak reading the strongest. A restart clears it for a while, but the real fix is a rollback. If no event shows up at all, you're in the fourth reading, and Dynatrace can't help you further: the change happened somewhere it doesn't observe.

That last case is common, and it's the reason "validate the candidate" stays a manual step even on the best-instrumented environment.

Three signals that matter beyond the problem feed

Problems cover anomalies Davis can causally explain. Three signal classes deserve their own attention, because they degrade quietly or fire outside the problem model's sweet spot:

01Budget

SLO burn

Tells you how much more failure the service can afford this month.

02Outside-in

Synthetic failures

Expired certificates, broken DNS and third-party outages your telemetry never sees.

03Trajectory

Host trends

Disks and memory that creep for weeks without ever looking anomalous.

Three signals that degrade quietly and rarely page with the right urgency.

Service-level objectives. SLOs in Dynatrace track error-budget burn: the difference between "an incident happened" and "we can no longer afford incidents this month." Check the Service-Level Objectives app for anything trending toward breach. Burn rate is a decision signal. It should change how aggressively you respond to the next problem on that service, which is exactly the kind of judgment no threshold encodes.

Synthetic failures. Synthetic monitors (browser clickpaths and HTTP checks) are your outside-in view. They catch expired certificates, broken DNS and third-party failures that produce zero anomalies in your own telemetry. Review failure patterns under Synthetic and make sure global outage criteria (how many locations must fail, for how long) match what you'd actually page on. The default may be noisier or laxer than your reality.

Host health trends. Slow burns (a disk filling at a steady 2% a day, memory creeping across weeks) are trajectories, not anomalies. Davis raises predictive disk-full events for some of these, but a periodic saturation review of Hosts, or a scheduled DQL query over dt.host.disk.used.percent, catches the rest before they become 2 a.m. problems.

None of these pages anyone with useful urgency by default. They're the signals a weekly review is supposed to catch, and the review is the first thing skipped in a busy sprint.

Time to detection is solved. Time to action isn't.

Here's the sober arithmetic. On a well-instrumented Dynatrace environment, detection takes minutes at worst, often under two. But the incident clock doesn't stop at detection:

01Detect

90 seconds

Davis opens the problem with a root-cause candidate attached.

02Pick up

Page and wake

Someone reads the card after the page and the context switch.

03Validate

Cross-check

Confirm the candidate against changes Dynatrace never observed.

04Act

Fix and verify

Restart, roll back or scale, then confirm the symptoms cleared.

Detection is the first step and the fastest. The other three wait on a person.

Pick up the problem card, after the page, the wake-up and the context switch.
Validate the root-cause candidate. Davis is good, but it can only reason over what it observes. Was there a config change in your cloud account? A deploy from a pipeline Dynatrace isn't watching? A quota change on the database service? That correlation is manual.
Decide and act. Restart, roll back, scale or fail over, under change-management rules, usually from a different console than the one showing the problem.
Verify and close. Confirm symptoms cleared and write up what happened.

If detection takes 90 seconds and the rest takes four hours, detection is under 1% of incident duration. Every additional euro spent sharpening detection optimizes the fraction that's already solved. The other 99%, the action gap, is organizational: problems wait in queues, queues wait for people, and people have working hours, on-call fatigue and competing pages. Dynatrace hands you the best problem card in the industry, and then the card sits there.

Where this series goes next

Closing that gap is an engineering project with two routes. You can build it yourself from Dynatrace's native pieces (problem notification webhooks, the Problems API, DQL enrichment and AutomationEngine workflows), which is what part two walks through, including where the ceiling is. Or you can put an action layer on top of the problem feed, which is part three.

That's the gap CloudThinker closes. It connects to Dynatrace read-only, subscribes to the Problems feed, enriches each problem with DQL context plus what Dynatrace can't see (recent config changes, cloud-side state), validates the Davis root-cause candidate, and proposes a fix. It acts within Auto Mode, so you decide what runs on its own and what waits for approval, and escalation stays intact.

Try CloudThinker free with 100 premium credits and no card required, or continue with the native-tools automation guide.