New Relic Automation: The APM Signals Actually Worth Alerting On

There's a New Relic account out there — maybe yours — with 400 alert conditions across 60 policies. And there are roughly six NRQL queries that actually describe whether the business is healthy. The distance between those two numbers is where on-call rotations go to die.

This is the core problem of New Relic automation: the platform collects everything — traces, transactions, spans, entity relationships, deployment markers — but turning that data into action still depends on a human reading a notification at 2 a.m. and deciding what to do. Before you can automate any of that, you need to know which signals deserve automation in the first place. Most accounts alert on the wrong ones.

This guide maps the landscape: how alert condition sprawl happens, why static thresholds fail on dynamic services, the four APM signals that actually predict incidents (with the NRQL behind each), the entity data nobody queries during triage, and what all of this costs in on-call hours.

This is part one of a three-part series. Part two is a hands-on guide to alert automation with New Relic's native tools — NRQL conditions, workflows, webhooks, NerdGraph. Part three covers putting an AI agent on the other end of your New Relic alerts.

How you end up with 400 alert conditions

Nobody designs alert sprawl. It accretes.

A new service ships, and the team copies the alert policy from the last service — thresholds included, whether or not they fit. An incident happens, and the postmortem action item is "add an alert for this," so a condition gets bolted on for a failure mode that will never recur in that exact shape. A team dissolves, and its policies outlive it, paging a rotation that no longer knows what the service does. Deleting a condition feels risky; adding one feels responsible. The ratchet only turns one way.

The result is a portfolio where a handful of conditions produce most of the noise. Find them:

SELECT count(*) FROM NrAiIncident
WHERE event = 'open'
FACET conditionName, policyName
SINCE 30 days ago LIMIT 25

Run that in the query builder and read the top of the list. In most accounts, the top five conditions account for well over half of all incidents opened. Now ask the on-call engineers what they did the last ten times each one fired. If the answer is "checked the dashboard, it was fine, closed it," that condition is not an alert — it's a subscription to interruptions.

Static thresholds on dynamic services

The second structural problem: most conditions are static thresholds ("error rate above 5%", "response time above 500 ms") applied to services whose normal behavior isn't static.

A batch-heavy API has a different latency profile at 3 a.m. than at 3 p.m. A B2B product's traffic drops 80% on weekends. A static threshold tight enough to catch a real regression during peak hours will false-positive every night; a threshold loose enough to stay quiet at night will sleep through a daytime incident. Teams resolve this tension in the worst possible way — they loosen the threshold until it stops paging, at which point it stops detecting.

New Relic's answer is anomaly detection: baseline conditions that model the metric's recent behavior and alert on deviation rather than absolute value. It's genuinely useful for signals with strong periodicity — throughput and latency especially. It is not a cure-all: baselines chase gradual degradation (a service that gets 2% slower every week never deviates enough to fire), and they need a few days of data before they're trustworthy. The practical pattern is anomaly conditions for throughput and latency, static conditions for things with a true hard limit — error budgets, queue depth, saturation.

The deeper fix, though, is upstream of the threshold type: alert on fewer, better signals.

The four APM signals that predict real incidents

Strip away the sprawl and most production incidents on an APM-instrumented service announce themselves through four signals. Each one below comes with the NRQL that describes it — these are the queries meaningful alert conditions get built on, and the queries a competent responder runs first.

1. Error rate, broken down by transaction

Aggregate error rate hides more than it reveals — a 2% overall rate could be noise spread evenly, or a checkout endpoint failing 40% of the time. Always facet:

SELECT percentage(count(*), WHERE error IS true) AS 'error rate'
FROM Transaction
WHERE appName = 'checkout-service'
FACET name
SINCE 30 minutes ago

An alert on the faceted version catches the broken endpoint while the blended number still looks fine. This is the single highest-value condition most accounts don't have.

2. Latency percentiles, not averages

Average duration is the most misleading number in APM. A p50 of 80 ms with a p99 of 4 seconds means your unluckiest 1% of requests — often your biggest customers, with the most data — are having a terrible time, and the average will never tell you.

SELECT percentile(duration, 50, 95, 99)
FROM Transaction
WHERE appName = 'checkout-service'
TIMESERIES SINCE 3 hours ago

Apdex compresses the same idea into a single satisfaction score if you've tuned the threshold per service (the default 0.5 s is wrong for almost everyone). Percentiles are more honest; alert on p95 or p99, and keep p50 on the dashboard for contrast.

3. Throughput shifts — in both directions

A traffic spike is obvious. A traffic drop is the one that gets missed — and a sudden 60% throughput drop usually means something upstream broke: a load balancer, a DNS change, a mobile release that can't reach you. Compare against the same window last week to control for time-of-day:

SELECT rate(count(*), 1 minute)
FROM Transaction
WHERE appName = 'checkout-service'
SINCE 1 hour ago COMPARE WITH 1 week ago

This is the signal where anomaly detection earns its keep, because "normal" throughput is a moving target that a static floor can't track.

4. External service degradation

Your service is often slow because someone else's service is slow — a payment provider, a third-party API, an internal dependency two teams away. The agent measures time spent in external calls on every transaction:

SELECT average(externalDuration), average(duration)
FROM Transaction
WHERE appName = 'checkout-service'
TIMESERIES SINCE 3 hours ago

When externalDuration rises and the gap between it and total duration stays flat, the problem is not in your code — and knowing that in the first two minutes changes the entire response. It's the difference between paging your own team and opening a status-page tab for your vendor.

The triage data nobody uses: entity relationships

Here's the strange part. New Relic already knows your topology — which services call which, which hosts run which applications, which alerts belong to which entities. Every instrumented component is an entity with a GUID and a graph of relationships, queryable through NerdGraph:

{
  actor {
    entity(guid: "YOUR_ENTITY_GUID") {
      name
      relatedEntities {
        results {
          type
          target { entity { name entityType alertSeverity } }
        }
      }
    }
  }
}

That query answers the question every responder actually has at 2 a.m. — what does this service depend on, and are any of those dependencies also alerting? — in one round trip. Almost nobody runs it. Instead, triage happens by tribal knowledge: the senior engineer who "just knows" the checkout service falls over when the pricing service is slow. When that engineer is asleep or gone, the knowledge is gone, and the mean time to understanding doubles even though all the data was sitting in the entity graph the whole time.

Any serious attempt at New Relic automation has to use this graph. Alert-to-entity-to-neighbors is the traversal that turns "an alert fired" into "here's the blast radius."

What alert fatigue actually costs

Put sober numbers on it. Suppose your top noisy conditions generate 300 incidents a month and 80% are non-actionable — figures well within the normal range for an untuned account. If each false alarm costs 10 minutes of an engineer's attention (acknowledge, glance at a dashboard, decide it's nothing, try to refocus), that's 40 hours a month — a full week of engineering time spent confirming that nothing is wrong.

The second-order cost is worse: after enough false alarms, response latency to real alerts stretches, because the prior on any given page being noise is 4-to-1. The alert that mattered gets the same weary 10-minute glance as the 40 that didn't. Alert fatigue doesn't just waste hours; it degrades the one thing the alerting system exists to protect.

From signal to action

Knowing the right signals is the foundation, not the finish line. The gap that remains is the one this series is about: an alert fires, and everything after that — running the error breakdown, checking percentiles around the incident window, looking for a deployment marker, walking the entity graph — is still manual, still 2 a.m., still dependent on who happens to be on call.

Part two closes part of that gap with New Relic's own machinery: NRQL alert conditions done properly, workflows, muting rules, webhook payloads, and NerdGraph automation. Read it before buying anything — the native tools go further than most teams push them.

When the investigation itself gets automated

The four queries above are exactly what CloudThinker agents run when a New Relic alert fires — error breakdown, percentiles, throughput comparison, dependency check — through a read-only connection, before a human ever opens a dashboard. The agent arrives at the incident with the evidence already gathered, and nothing changes in your systems without the autonomy level you've configured. Part three covers it in depth, or try CloudThinker free — 100 premium credits, no card required.