New Relic Automation: The APM Signals Actually Worth Alerting On

There's a New Relic account out there, maybe yours, with 400 alert conditions across 60 policies. When checkout-service pages at 2 a.m., the on-call opens the APM summary, then the errors inbox, then distributed traces, then the deployments page, in that order, every time. The condition said something is wrong. The next half hour of clicking is how they find out what. And somewhere in that account there are roughly six NRQL queries that actually describe whether the business is healthy. The distance between 400 and six is where on-call rotations go to die.

That's the core problem of New Relic automation. The platform collects everything (traces, transactions, spans, entity relationships, deployment markers), but turning that data into action still depends on a person reading a notification at 2 a.m. and deciding what to do. Before you automate any of that, you need to know which signals deserve automation in the first place. Most accounts alert on the wrong ones, and automating the wrong signal only delivers noise faster.

This guide maps the landscape: how alert condition sprawl happens, why static thresholds fail on dynamic services, the four APM signals that actually predict incidents (with the NRQL behind each), the entity data nobody queries during triage, and what all of it costs in on-call hours. Every section has a query you can run today.

This is part one of a three-part series. Part two is a hands-on guide to alert automation with New Relic's native tools: NRQL conditions, workflows, webhooks and NerdGraph. Part three covers putting an AI agent on the other end of your New Relic alerts.

The patterns below are structural. None of them is "New Relic is missing a feature." They're what happens to any alerting estate that grows by addition and never by review.

01Estate

Condition sprawl

Hundreds of conditions, a handful of which open most incidents.

02Threshold

Static on dynamic

One number applied to traffic that changes shape every hour.

03Signal

Blended metrics

Averages and service-wide rates that hide the broken endpoint.

04Topology

Unused entity graph

New Relic knows what depends on what. Triage runs on memory instead.

---

1. How you end up with 400 alert conditions

Nobody designs alert sprawl. It accretes.

A new service ships, and the team copies the alert policy from the last service, thresholds included, whether or not they fit. An incident happens, and the postmortem action item is "add an alert for this," so a condition gets bolted on for a failure mode that will never recur in that exact shape. A team dissolves, and its policies outlive it, paging a rotation that no longer knows what the service does. Deleting a condition feels risky. Adding one feels responsible. The ratchet only turns one way.

How to detect it. New Relic records every incident lifecycle event in NrAiIncident. A handful of conditions usually produce most of the noise, so find them:

SELECT count(*) FROM NrAiIncident
WHERE event = 'open'
FACET conditionName, policyName
SINCE 30 days ago LIMIT 25

Run that in the query builder and read the top of the list. Then check how long those incidents stayed open. Incidents that close on their own within a few minutes are the classic flapping signature:

SELECT count(*), average(durationSeconds) FROM NrAiIncident
WHERE event = 'close'
FACET conditionName
SINCE 30 days ago LIMIT 25

Typical impact. In most accounts, the top five conditions account for well over half of all incidents opened. Ask the on-call what they did the last ten times each one fired. If the answer is "checked the dashboard, it was fine, closed it," that condition isn't an alert. It's a subscription to interruptions.

2. Static thresholds on dynamic services

Most conditions are static thresholds ("error rate above 5%", "response time above 500 ms") applied to services whose normal behavior isn't static.

A batch-heavy API has a different latency profile at 3 a.m. than at 3 p.m. A B2B product's traffic drops 80% on weekends. A threshold tight enough to catch a real regression at peak will false-positive every night. A threshold loose enough to stay quiet at night will sleep through a daytime incident. Teams resolve this in the worst possible way: they loosen the threshold until it stops paging, at which point it stops detecting.

How to detect it. Plot the signal a condition watches against its own threshold over a full week, and look at the shape:

SELECT percentile(duration, 95)
FROM Transaction
WHERE appName = 'checkout-service'
TIMESERIES 1 hour SINCE 7 days ago

If the line has a strong daily or weekly rhythm and the threshold is a flat line through it, the condition is either noisy at the peaks or blind in the troughs. Usually both, at different times.

New Relic's answer is anomaly detection: conditions that model the metric's recent behavior and alert on deviation rather than absolute value. It's useful for signals with strong periodicity, throughput and latency especially. It isn't a cure-all. Baselines chase gradual degradation (a service that gets 2% slower every week never deviates enough to fire), and they need a few days of data before they're trustworthy. The practical split: anomaly conditions for throughput and latency, static conditions for things with a true hard limit, like error budgets, queue depth and saturation.

The deeper fix, though, sits upstream of the threshold type: alert on fewer, better signals.

3. The four APM signals that predict real incidents

Strip away the sprawl and most production incidents on an APM-instrumented service announce themselves through four signals. Each comes with the NRQL that describes it. These are the queries good alert conditions get built on, and the queries a competent responder runs first.

01Errors

Rate by transaction

Faceted by name, so one failing endpoint shows while the blend looks fine.

02Latency

p95 and p99

Percentiles, not averages. The slow tail is often your biggest customers.

03Throughput

Shifts both ways

A sudden drop usually means something upstream broke before you.

04Dependencies

External duration

Time spent waiting on someone else, separated from your own code.

Error rate, broken down by transaction

Aggregate error rate hides more than it shows. A 2% overall rate could be noise spread evenly, or a checkout endpoint failing 40% of the time. Always facet:

SELECT percentage(count(*), WHERE error IS true) AS 'error rate'
FROM Transaction
WHERE appName = 'checkout-service'
FACET name
SINCE 30 minutes ago

An alert on the faceted version catches the broken endpoint while the blended number still looks fine. This is the single highest-value condition most accounts don't have.

Latency percentiles, not averages

Average duration is the most misleading number in APM. A p50 of 80 ms with a p99 of 4 seconds means your unluckiest 1% of requests (often your biggest customers, with the most data) are having a terrible time, and the average will never tell you.

SELECT percentile(duration, 50, 95, 99)
FROM Transaction
WHERE appName = 'checkout-service'
TIMESERIES SINCE 3 hours ago

Apdex compresses the same idea into a single satisfaction score if you've tuned the threshold per service (the default 0.5 s is wrong for almost everyone). Percentiles are more honest. Alert on p95 or p99, and keep p50 on the dashboard for contrast.

Throughput shifts, in both directions

A traffic spike is obvious. A traffic drop is the one that gets missed, and a sudden 60% drop usually means something upstream broke: a load balancer, a DNS change, a mobile release that can't reach you. Compare against the same window last week to control for time of day:

SELECT rate(count(*), 1 minute)
FROM Transaction
WHERE appName = 'checkout-service'
SINCE 1 hour ago COMPARE WITH 1 week ago

This is the signal where anomaly detection earns its keep, because "normal" throughput is a moving target a static floor can't track.

External service degradation

Your service is often slow because someone else's service is slow: a payment provider, a third-party API, an internal dependency two teams away. The APM agent measures time spent in external calls on every transaction:

SELECT average(externalDuration), average(duration)
FROM Transaction
WHERE appName = 'checkout-service'
TIMESERIES SINCE 3 hours ago

When externalDuration rises and the gap between it and total duration stays flat, the problem isn't in your code. Knowing that in the first two minutes changes the whole response. It's the difference between paging your own team and opening a status-page tab for your vendor.

4. The triage data nobody uses: entity relationships

Here's the strange part. New Relic already knows your topology: which services call which, which hosts run which applications, which alerts belong to which entities. Every instrumented component is an entity with a GUID and a graph of relationships, queryable through NerdGraph:

{
  actor {
    entity(guid: "YOUR_ENTITY_GUID") {
      name
      relatedEntities {
        results {
          type
          target {
            entity {
              name
              entityType
              alertSeverity
            }
          }
        }
      }
    }
  }
}

That query answers the question every responder actually has at 2 a.m. (what does this service depend on, and are any of those dependencies also alerting?) in one round trip. Almost nobody runs it. Triage runs on tribal knowledge instead: the senior engineer who "just knows" checkout falls over when the pricing service is slow. When that engineer is asleep or gone, the knowledge is gone, and the time to understanding doubles even though the data was in the entity graph the whole time.

How to detect the gap. Ask two questions after your next incident. Did anyone look at related entities before forming a hypothesis? And did the first person paged know which upstream service turned out to be the cause? If the answers are "no" and "only because they'd seen it before," your triage depends on memory.

Any serious attempt at New Relic automation has to use this graph. Alert to entity to neighbors is the traversal that turns "an alert fired" into "here's the blast radius."

5. What alert fatigue actually costs

Put sober numbers on it. These are illustrative, but well within the normal range for an untuned account. Say your noisy conditions open 300 incidents a month and 80% are non-actionable. If each false alarm costs 10 minutes of an engineer's attention (acknowledge, glance at a dashboard, decide it's nothing, try to refocus), that's 40 hours a month: a full week of engineering time spent confirming that nothing is wrong.

The second-order cost is worse.

01Noise

300 incidents

Untuned conditions open hundreds of incidents a month, most of them nothing.

02Habit

Glance, close

On-call learns that a page usually means checking a dashboard and moving on.

03Prior

4 to 1 noise

Every new page starts with the assumption that it is probably nothing.

04Cost

Real alert waits

The page that mattered gets the same tired ten-minute glance.

After enough false alarms, response to real alerts slows down, because the prior on any page being noise is 4 to 1. The alert that mattered gets the same weary glance as the 40 that didn't. Fatigue doesn't only waste hours. It degrades the one thing the alerting system exists to protect.

---

What good New Relic alerting looks like

The healthy end state is boring and specific:

• Policies follow ownership. A checkout-team policy pages the checkout on-call. No global latency policy that pages everyone.
• Paging conditions watch symptoms. Error rate by transaction, p95 or p99 latency, throughput against last week, external duration. Cause-level conditions go to a channel or a ticket.
• Anomaly conditions where traffic has a rhythm, static ones where there's a hard limit.
• Loss of signal is configured on anything whose disappearance matters, so a dead service doesn't look like a healthy one.
• Every condition has an owner and a review date, and the top-noise query from section 1 gets run monthly.

Why the cleanup doesn't stick

Most teams have done an alert cleanup. It works for about six weeks. Then two services ship with copied policies, a postmortem adds three conditions, traffic grows into a threshold set last spring, and the estate quietly grows back.

Hygiene is a rate problem, not a project. And even a clean estate still hands the on-call a symptom and wishes them luck. Running the error breakdown, checking percentiles around the incident window, looking for a deployment marker, walking the entity graph: all of it stays manual, at 2 a.m., and depends on who happens to be on call.

Where this series goes next

Part two closes part of that gap with New Relic's own tooling: NRQL alert conditions done properly, workflows, muting rules, webhook payloads and NerdGraph automation. Read it before buying anything. The native tools go further than most teams push them.

The rest of the gap is the investigation itself. The four queries above are exactly what CloudThinker runs when a New Relic alert fires (error breakdown, percentiles, throughput comparison, dependency check) through a read-only connection, before a person opens a dashboard. It arrives at the incident with the evidence gathered, and acts within Auto Mode, so you decide what runs on its own and what waits for approval. Part three covers it in depth.

If you want to see it on your own account first, try this in your first session:

"Which New Relic alert conditions opened the most incidents in the last 30 days, how many of those closed on their own within ten minutes, and which services report APM data with no condition on error rate or latency?"

Try CloudThinker free with 100 premium credits and no card required, or continue with part two.