New Relic Automation with AI Agents: From Alert to Root Cause

The alert says the error rate on checkout-service doubled at 14:30 UTC. That's one sentence of information. Between that sentence and knowing why sit roughly twenty NRQL queries: errors faceted by transaction, then by error class, latency percentiles for the hour around the spike, throughput against last week, external service durations, and finally the question everyone should have asked first: did anything deploy? An experienced SRE runs that sequence in twenty to forty minutes. The condition that fired took milliseconds. Everything expensive happens after it.

This last part of the series is about closing that gap. Part one mapped the problem: alert condition sprawl, static thresholds on dynamic services, and the APM signals that actually predict incidents. Part two built everything New Relic's native tooling offers (NRQL conditions, policies, workflows, webhooks with custom payloads, muting rules and NerdGraph condition management) and ended at the honest ceiling: workflows route incidents, they don't read the APM data for you.

CloudThinker is the layer that reads it. Your conditions, dashboards and instrumentation stay where they are. This isn't a New Relic replacement. CloudThinker connects read-only, picks up issues as they open, runs the investigation an SRE would run, and proposes the fix (or, under a policy you wrote, applies it).

---

Why the investigation never got automated

Part two's enrichment attaches one NRQL query to the page. That's the right instinct with a structural limit: real triage isn't one query, it's a chain where each answer decides the next. If 94% of the new errors are Net::ReadTimeout on one transaction, the next query is about the external call. If errors are spread evenly across every transaction, the next query is about hosts and the runtime. If they started two minutes after a deployment marker, the next query is about what that release touched.

01If errors spread evenly

Runtime or host

Next query: latency and errors faceted by host, then host metrics.

02If one call times out

Dependency

Next query: externalDuration against duration, then the upstream entity.

03If errors start at a marker

Release

Next query: transactions the release touched, and the rollback path.

A workflow can only follow branches someone drew before the incident. Writing every branch for every service is a full-time job that's out of date by the next release. So the chain stays in a person's head, and it runs at the speed of a tired person clicking between tabs.

---

Connecting New Relic: a user API key, about five minutes

01Observability

New Relic

• Incidents and issuesNrAiIncident, alert conditions, policies
• APM telemetryTransaction, TransactionError, Span
• Entities and changesRelated entities, deployment markers

User key on a read-only user

02Investigation

CloudThinker

• Pick up open issuesThe same issues your workflows route
• InvestigateNRQL through NerdGraph, query by query
• Propose the fixWith rollback steps and the evidence

No agents installed

03Where the team works

Chat and on-call

• Slack or TeamsFindings and approvals in the thread
• PagerDutySame escalation path, evidence attached
• JiraFollow-ups with the investigation linked

Nothing new to watch

The connection needs three things: a user API key, your account ID, and your data center region (US or EU). Create the key from your user menu → API keys in New Relic. Better: create a dedicated user assigned a read-only role and generate the key from that user, so the key can never do more than its owner can. New Relic's API key documentation covers the key types. The user key is the one that works with NerdGraph, which is how CloudThinker queries everything.

Paste the key and account ID into CloudThinker and the first scan starts. No agent installs, no changes to your instrumentation, no write access at connection time. The connections guide has the step-by-step, including the recommended read-only role setup for your security review.

---

The investigation, run the way an SRE would run it

01Errors

Break down

Which transactions, which error classes, how concentrated.

02Shape

Frame the window

Percentiles and throughput against the same hour last week.

03Changes

Correlate

Deployment markers on the entity and everything related to it.

04Output

Finding

Likely cause with the query results, a fix and a rollback note.

When a condition opens an incident, CloudThinker starts with the same NRQL an on-call engineer would type. It starts within seconds, and it doesn't lose its place between tabs. Here's the 14:30 error spike on checkout-service, worked hypothesis first.

Break the errors down. Not "errors are up" but which transactions, which error classes:

SELECT count(*) FROM TransactionError
WHERE appName = 'checkout-service'
FACET transactionName, `error.class`
SINCE 30 minutes ago

This one query splits the readings. Errors concentrated on one transaction and one class point at a dependency or a code path. Errors spread across every transaction point at the runtime or the infrastructure underneath, a different path entirely. Here it came back with 94% Net::ReadTimeout on WebTransaction/Controller/checkout/payment. The investigation is already half over.

Frame the window. Averages hide bimodal failures. Percentiles show whether everything got slow or a subset fell off a cliff:

SELECT percentile(duration, 50, 95, 99) FROM Transaction
WHERE appName = 'checkout-service'
TIMESERIES 5 minutes SINCE 2 hours ago

It also compares throughput against the same window a week earlier (COMPARE WITH 1 week ago), because "error rate doubled" during a traffic drop means something very different from during a spike. Here p50 was flat and p99 had jumped to the payment timeout, with throughput normal. A subset fell off a cliff.

Check what changed, and where. Most production incidents trace back to a deployment or a configuration change, and it's the check people run last because it lives in a different tab. The timeouts say the payment call is slow, so the next step walks the entity graph from part one to the services checkout-service depends on, then pulls change-tracking markers for each:

SELECT * FROM Deployment
WHERE entity.guid IN ('CHECKOUT_GUID', 'PAYMENTS_GATEWAY_GUID')
SINCE 2 hours ago

A deploy to payments-gateway at 14:28, two minutes before the spike, is the likely cause. Nobody had to remember the service topology to find it. The graph held it.

Write down the answer. Not a forwarded alert, a finding: what fired, the error breakdown, the latency shape, the correlated change, the most likely cause with the query results attached, and a proposed remediation (roll back the 14:28 payments-gateway release) with a rollback note. It lands in the Slack channel your part-two workflow already posts to, as a diagnosis instead of a link.

The difference from part two is the branching. A workflow executes the steps you predicted. The agent picks the next query based on what the last one returned (timeouts lead to dependency checks, uniform errors lead to infrastructure checks), the way an engineer reasons, with the reasoning logged.

---

Acting on the finding

Reads (NRQL, entities, change markers) are how the investigation works. Writes are things like adjusting a threshold, adding a muting rule, retiring a flappy condition or rolling back a release. CloudThinker acts within Auto Mode, so you decide what runs on its own and what waits for approval.

Escalation stays intact. If the evidence says real outage rather than noise, CloudThinker pages through your existing path with the investigation attached. Every query run, finding, approval and change lands in an audit trail.

---

What a first scan typically finds

Alongside live incident response, the first connection runs a hygiene pass over your alerting estate: the review from parts one and two, compressed into minutes. The numbers below are illustrative, a composite for a mid-market environment with 30 to 60 instrumented services and a few hundred alert conditions. Yours will differ.

Look at the shape rather than the numbers. Almost none of these rows say "add more monitoring." They say your existing alerting is misleading you in specific, fixable ways. Each row comes with a proposed fix, most of them one NerdGraph mutation away, and each one goes through Auto Mode.

---

Why this beats the traditional setup

The deeper difference is where judgment lives. Native automation encodes judgment you had last quarter. An investigation layer applies it to tonight's evidence and shows its work, so you can check it before you act on it.

---

Prompts to try in your first session

CloudThinker is conversational. You ask in plain language, and the answer cites the NRQL and results so you can verify rather than trust:

"The error rate on checkout-service doubled around 14:30 UTC. Break the errors down by transaction and error class, pull latency percentiles for the two hours around it, and check for deployments on this service and anything it depends on. Propose only, change nothing."

"Which alert conditions opened more than ten incidents last week that nobody acted on, and what threshold or aggregation-window changes would quiet them without losing real signal?"

"List every service reporting APM data that has no alert condition on error rate or latency, ranked by throughput."

Run the first one during your next real incident and compare it with what on-call assembled by hand. That comparison, not this article, should decide whether CloudThinker earns a place in your escalation path.

---

What CloudThinker won't do

An AI agent connected to production telemetry should raise questions. The answers:

• Read-only by default. The connection key reads. Changing anything in New Relic (thresholds, muting rules, workflow routing) needs write permission you grant, and then goes through Auto Mode.
• No silent changes. Every change is proposed with its evidence and logged with who approved it.
• No skipped escalations. CloudThinker never decides an incident isn't worth waking someone for. Suppression is yours to configure, explicit and logged.
• No unauditable actions. Every query, finding, proposal and change is in the audit trail with its evidence.

If you built the part-two automation, keep it. Webhooks and workflows are the right tool for deterministic routing. Put CloudThinker on the part that was never automatable: reading the data and deciding what it means.

Closing the loop

Part one showed which APM signals matter and why static alerting drowns them. Part two took native automation to its ceiling: incidents routed, enriched and correlated, and still uninvestigated. The twenty NRQL queries between "error rate doubled" and "the 14:28 deploy to the payments gateway caused it" were the part of on-call that only a person could do. Teams that put an investigation layer there typically cut time to diagnosis from half an hour to a few minutes, and clean up the part-one hygiene debt along the way.

Try CloudThinker free with 100 premium credits and no card required, and follow the connections guide to see your next incident arrive already diagnosed.