New Relic Alert Automation with NRQL, Workflows, and NerdGraph

The p95 latency condition on checkout-api fires at 2:47 a.m. Applied Intelligence has already folded the three related incidents into one issue. The workflow routes it to #checkout-oncall with an enrichment attached: the top error classes for the last 30 minutes. Everything worked exactly as configured. Then the on-call opens six tabs (the APM summary, the errors inbox, distributed traces, the deployments page, the infrastructure host, and the dashboard someone built two years ago) because the pipeline that detected and routed the problem can't investigate it.

That's a fair picture of how far New Relic's native alerting goes, and it's further than most teams take it. Well-written NRQL conditions, sane policies, workflows with enrichments, muting rules, correlation and NerdGraph can turn a noisy estate into a pipeline that pages less and pages with context. All of it is built from features you already pay for. The catch is where the pipeline ends.

In part one of this series we covered which APM signals deserve alerts in the first place (error rate, latency percentiles, throughput shifts, external service degradation) and the NRQL behind each. Here we turn those queries into a production alerting system, tool by tool, using nothing outside the platform. Budget half a day for a first serious pass on one team's services. Part three covers the layer that sits on top.

Build the stack in this order. Routing a noisy condition only delivers noise faster, so the trigger comes first and code management comes last.

01Trigger

NRQL conditions

Right window, method, duration and loss of signal, so a breach means something.

02Group

Policies

Incident preference decides how incidents roll up into issues.

03Route

Workflows

Filter issues to destinations, with one enrichment query attached.

04Silence

Muting rules

Scheduled, tightly scoped quiet for known windows.

05Manage

NerdGraph

Every condition as code, reviewed and versioned.

---

Step 1: NRQL alert conditions, get the signal math right

Every alert worth having on New Relic is a NRQL alert condition: a query evaluated on a rolling window and compared against a threshold. Console path: Alerts → Alert conditions → New alert condition → Write your own query.

Start from the part-one queries:

-- Error rate for one service
SELECT percentage(count(*), WHERE error IS true)
FROM Transaction
WHERE appName = 'checkout-api'

-- p95 latency, seconds
SELECT percentile(duration, 95)
FROM Transaction
WHERE appName = 'checkout-api'

The query is the easy half. Three settings decide whether the condition is trustworthy.

Aggregation window and method. The window (default 60 seconds) is the bucket each data point is computed over. The method decides when a bucket counts as complete. EVENT_FLOW (the default) closes a bucket when newer data arrives, which is right for steady, high-throughput telemetry like APM transactions. EVENT_TIMER closes a bucket a fixed time after its last data point. Use it for sparse or batchy data, such as cloud integration metrics or infrequent log events, where event flow can evaluate a bucket before late data lands and fire falsely. If a condition on a low-traffic queue keeps flapping, this setting is usually why.

Threshold duration. "Above 5% at least once in 5 minutes" pages you for a single bad bucket. "Above 5% for at least 5 minutes" waits out blips. For paging conditions, prefer for at least, sized to your real tolerance. Keep a warning threshold below the critical one so trends surface before they page.

Loss of signal. A crashed service sends no data, and a condition on missing data evaluates nothing, so it stays silently green. Set the signal-loss expiration (10 minutes, say) and choose open a new incident on signal loss for anything whose disappearance matters. This is the difference between "the error rate is fine" and "the service is gone."

For dynamic services where any static threshold is wrong half the day, use an anomaly condition instead. New Relic builds a baseline from recent behavior and you alert on deviation. It's good for throughput and latency on traffic with a strong daily shape, and still wrong on the day of a marketing launch, which is what muting rules are for (step 4).

Step 2: Policies and incident preference, control the blast radius

Conditions live in alert policies, and each policy carries one deceptively important setting: incident preference, which controls how incidents roll up into issues (the things that actually notify).

• One issue per policy (default): everything in the policy, one issue at a time. Quiet, but a new failure gets swallowed by an issue that's already open for something else.
• One issue per condition: each condition gets its own issue. The sane default for most teams.
• One issue per condition and signal: every entity or facet gets its own issue. Maximum granularity, and maximum noise when 40 hosts degrade together.

Organize policies by ownership, not by technology. A checkout-team policy that pages the checkout on-call beats a global latency policy that pages everyone. Condition sprawl (hundreds of conditions across dozens of policies nobody remembers creating) starts with policies organized any other way.

Step 3: Workflows and destinations, routing with context

Notification routing is two objects. Destinations are a configured Slack workspace, PagerDuty service, email, Jira project, ServiceNow instance or generic webhook. Workflows filter issues and send them to destinations. Console: Alerts → Workflows → Add a workflow.

Two workflow features do real work.

Enrichments. A workflow can run a NRQL query when an issue fires and attach the result to the notification, for example the current error breakdown for the affected service:

SELECT count(*) FROM TransactionError
WHERE appName = 'checkout-api'
FACET `error.class` SINCE 30 minutes ago

This is the closest native New Relic gets to automated investigation: one predefined query, attached to the page. Use it. An on-call engineer who sees the top error class before opening the laptop is minutes ahead.

Webhook destinations with custom payloads. For anything downstream (a ticketing system, an automation runbook, your own service), send a webhook and shape the JSON with message templates (Handlebars-style variables):

{
  "issueId": {{ json issueId }},
  "title": {{ json annotations.title.[0] }},
  "priority": {{ json priority }},
  "state": {{ json state }},
  "policies": {{ json accumulations.policyName }},
  "conditions": {{ json accumulations.conditionName }},
  "entities": {{ json entitiesData.names }},
  "issueUrl": {{ json issuePageUrl }}
}

The state field moves through the issue lifecycle (ACTIVATED, ACKNOWLEDGED, CLOSED, among others). Handle every state you subscribe to in your receiver, or you'll open tickets you never close. This webhook is also the standard integration point for anything acting on New Relic alerts from outside, which matters in part three.

Step 4: Muting rules, schedule the silence

Every deploy window and maintenance job that pages someone teaches on-call to ignore pages. Muting rules (Alerts → Muting rules) suppress notifications for incidents matching your criteria (a policy, condition, entity or tag) on a one-off or recurring schedule.

Two habits keep them safe. Scope tightly: mute the batch-etl entity during its window, not the whole policy. And never create an unscheduled, open-ended rule, because that's how a real outage goes unnoticed for six hours. Incidents still open while muted. Only the notification is suppressed, so the history stays intact.

Step 5: Correlation, let decisions shrink the noise

New Relic's incident correlation ("decisions") merges related incidents into one issue based on shared entities, timing and learned patterns. A database failure and the four service-latency incidents it caused become one notification instead of five. It's on by default with built-in global decisions, and you can define your own logic under Alerts → Correlate → Decisions.

Correlation usually cuts notification volume noticeably. Be clear on what it does: it groups symptoms that belong together. It doesn't tell you which of the five correlated incidents is the cause and which are downstream.

Step 6: NerdGraph, manage conditions as code

Past a handful of conditions, click-ops stops scaling. Thresholds drift, naming diverges, and nobody knows which conditions are load-bearing. The NerdGraph API manages the whole alerting surface programmatically. Creating a static NRQL condition:

mutation {
  alertsNrqlConditionStaticCreate(
    accountId: 1234567
    policyId: "987654"
    condition: {
      name: "checkout-api p95 latency"
      enabled: true
      nrql: {
        query: "SELECT percentile(duration, 95) FROM Transaction WHERE appName = 'checkout-api'"
      }
      terms: [
        {
          threshold: 1.5
          thresholdDuration: 300
          thresholdOccurrences: ALL
          operator: ABOVE
          priority: CRITICAL
        }
      ]
      signal: {
        aggregationWindow: 60
        aggregationMethod: EVENT_FLOW
        aggregationDelay: 120
      }
      expiration: { expirationDuration: 600, openViolationOnExpiration: true }
    }
  ) {
    id
    name
  }
}

Every setting from step 1 is here: a 60-second window with event flow, "for at least 300 seconds" (thresholdOccurrences: ALL), and loss of signal after 600 seconds opening an incident. Run it in the NerdGraph explorer at https://api.newrelic.com/graphiql with a user API key.

The matching alertsNrqlConditionsSearch query lets you export every existing condition. Do that first and diff it against what you think you have. The gap is usually educational. If you'd rather not hand-write GraphQL, the New Relic Terraform provider wraps the same API. Either way you get review, versioning and consistent thresholds across services.

---

One alert through the native stack

Go back to the 2:47 a.m. p95 condition on checkout-api. Before anyone acts, there are three reasonable readings, and each predicts something different in the data.

01Reading 1

Bad deploy

Errors step up at a deployment marker, on one transaction or one release.

02Reading 2

Slow dependency

externalDuration climbs with total duration; timeout errors on the external call.

03Reading 3

Host saturation

The regression sits on one or two hosts while the rest look normal.

Each reading has a query that confirms or kills it.

A bad deploy shows as a step change at a deployment marker. Pull the change-tracking markers for the entity, then line them up against errors by transaction:

SELECT * FROM Deployment
WHERE entity.guid = 'YOUR_ENTITY_GUID'
SINCE 3 hours ago

SELECT count(*) FROM TransactionError
WHERE appName = 'checkout-api'
FACET name TIMESERIES SINCE 3 hours ago

If errors jump within minutes of a marker and concentrate on the transactions the release touched, you have your answer.

A slow dependency shows as external time rising with total time, and timeout errors on the external call:

SELECT average(externalDuration), average(duration)
FROM Transaction
WHERE appName = 'checkout-api'
TIMESERIES SINCE 3 hours ago

If the two lines rise together and the gap between them stays flat, the time is being spent waiting on someone else. The top error class in the enrichment (a wall of Net::ReadTimeout, say) points the same way.

Host saturation shows as the regression concentrated on a few hosts:

SELECT percentile(duration, 95)
FROM Transaction
WHERE appName = 'checkout-api'
FACET host TIMESERIES SINCE 3 hours ago

Two hosts at 4 seconds while the rest sit at 300 ms is a placement or noisy-neighbor problem, not an application one.

Here's the honest result. The enrichment attached one of these signals (the error classes) and none of the conclusions. On this night the answer was reading two, which is the common case: most latency incidents that aren't deploys are dependencies. No workflow you configured in advance knew which dependency, and the on-call ran the other queries by hand.

What each native tool solves, and what it leaves

The right-hand column has one pattern: native automation handles the part you understood well enough to write down. Anything that needs reading evidence and choosing the next query stays with the on-call.

---

The ceiling: workflows route incidents, they don't read the trace

Build all six steps and you have a good alerting system: trustworthy conditions, sane grouping, notifications with context, scheduled silence, correlated issues and configuration in git. Be honest about what you don't have.

01Workflows

Route, not read

The issue reaches the right channel. Nothing queries the trace.

02Enrichment

One fixed query

Real triage is a sequence where each answer picks the next query.

03Correlation

Groups, not concludes

Five incidents become one issue. Which one is the cause stays open.

04Config

Decays

Thresholds set in March are wrong by August, and nobody gets told.

Routing isn't investigation. When the p95 condition fires, the workflow delivers the issue. It doesn't break errors down by transaction, pull percentiles around the incident window, check for a deployment marker ten minutes earlier, or walk entity relationships to see that the payment service's upstream dependency degraded first. Enrichments attach one predefined query. Real triage is a sequence where each answer decides the next, and that sequence still runs in a person's head, across six tabs, at whatever hour the page landed.

Configuration decays. Services change shape, and thresholds set in March are wrong by August. NerdGraph makes updating conditions cheap. Noticing they need updating is still an engineer reading charts.

Correlation groups, it doesn't conclude. Five incidents in one issue beats five pages, but "which one is the cause" is manual work, every time.

Native tooling automates everything up to the moment understanding is required. That moment is the expensive one.

The approval question you'll meet again

The webhook in step 3 is where most teams first wire New Relic to something that acts: a runbook that restarts a worker, a script that scales a pool. That forces a decision you'll see again in part three: which actions run unattended and which wait for a person. In part three, CloudThinker acts within Auto Mode, so you decide what runs on its own and what waits for approval. If you've already decided which restarts are safe to run from a webhook, you've done half that work.

What comes next

The investigation itself (querying NRQL the way an SRE would, correlating across entities, naming the likely cause with evidence attached) is what CloudThinker adds on top of New Relic. It connects read-only in about five minutes, picks up the same issues your workflows route, and works the incident before a person opens a tab. That's part three of this series.

To see it against your own account first, ask something like this in your first session:

"The p95 latency condition on checkout-api fired last night. Check for deployment markers on it and its related entities, compare external duration with total duration, and break latency down by host for that window. Tell me which reading the evidence supports. Propose only, change nothing."

Try CloudThinker free with 100 premium credits and no card required, or continue with part three.