Automating AppDynamics Health Rule Responses with Native Tools

Alert & Respond gets you further than most teams realize. It gets you right up to the point where the remediation script attached to your restart policy becomes its own unmaintained service: running on a node someone rebuilt in March, doing something nobody remembers approving. Before you get there, it's worth knowing exactly how far the native stack goes.

Part one covered why AppDynamics health rules generate so much noise out of the box: untuned defaults, baselines that don't know what day it is, violation storms during deploys, and node-versus-tier scoping mistakes. This article is the fix, using only what ships with the Controller: baselines, schedules, action suppression, policies, actions, HTTP request templates and the REST APIs. Exact steps, real API calls, nothing to buy.

We'll work in layers, from "fire less often" to "do something when you fire":

01Layer 1

Tune rules

Seasonal baselines, evaluation windows, load floors and warm-up.

02Layer 2

Known windows

Schedules stop evaluation; action suppression stops the pages.

03Layer 3

Policies

Bind violations to diagnostics, notifications and remediation.

04Layer 4

HTTP actions

Templated webhooks into chat, paging and ticketing.

05Layer 5

Events API

Pull violations and config for your own tooling.

---

Before you start: an API client and a token

Layers 2 and 5 use the API, so set up access first. Create an API client under Settings (gear icon) → Administration → API Clients, give it a role with the permissions you need (read-only for audits; the alerting API's write calls need configure rights on the application), and exchange its secret for a token (API clients docs):

TOKEN=$(curl -s -X POST "https://<controller-host>/controller/api/oauth/access_token" \
  -H "Content-Type: application/x-www-form-urlencoded" \
  -d "grant_type=client_credentials&client_id=<client-name>@<account-name>&client_secret=<client-secret>" \
  | jq -r .access_token)

Use separate clients for read-only audits and for anything that writes, so the audit log tells you which was which.

Layer 1: Tune the health rules themselves

Most violation noise is a configuration problem, not an alerting-tool problem. Open Alert & Respond → Health Rules, select the application, and edit the rules doing the most damage (part one's triage list). Three settings matter most (health rules docs).

Point baseline conditions at the right trend

The default baseline is a Daily Trend over the last 30 days. It knows that 9 a.m. is busier than 3 a.m., but it treats Tuesday 9 a.m. like every other 9 a.m. If your traffic has weekly shape (Monday-morning login spikes, a Tuesday report job, Friday batch runs), that comparison is wrong once a week and the rule fires on schedule.

Fix: create a seasonal baseline under Configuration → Baselines with a Weekly Trend pattern and a rolling time range, then edit the health rule condition to evaluate against that baseline instead of the default. Now Tuesday 9 a.m. is compared with previous Tuesdays at 9 a.m.

Widen the evaluation window on flapping rules

Each health rule evaluates against a trailing window of data, 30 minutes by default. For a rule that flaps (violates, resolves, violates again within the hour), the cheapest fix is a longer window in the rule's evaluation settings, so a two-minute latency blip can't drag the whole window over the line. For rules guarding fast-moving SLOs, shorten it instead and accept more sensitivity. Either way, decide per rule.

Add a load floor and respect warm-up

Baselines are unreliable at low load: 3 errors in 10 calls is a 30% error rate. AND a second condition onto the primary one, such as calls per minute greater than 50, the same pattern AppDynamics' own default rules use, so the rule can only violate when there's enough traffic to mean something.

For node-level rules, JVM warm-up after a deploy or restart looks exactly like a regression: cold caches, JIT compilation, connection pools filling. Handle that with deploy-scoped suppression in the next layer rather than loosening the rule for everyone.

Layer 2: Schedules and action suppression for known windows

Two different tools, often confused.

Health rule schedules stop evaluation during windows where the rule is meaningless. Create one under Alert & Respond → Schedules (for example "Business hours, Mon to Fri 08:00 to 20:00" in the application's time zone), then attach it in the health rule's settings. The nightly-batch CPU rule stops evaluating at night; the checkout-latency rule only runs when checkout traffic exists. Violations are never recorded, so dashboards stay clean too.

Action suppression stops actions while still recording violations. It's the right tool for maintenance and deploy windows, where you want the history but not the pages. Configure it under Alert & Respond → Actions → Action Suppression: name it, set the window, scope it to applications, tiers or nodes, and optionally disable agent reporting.

The part most teams miss: action suppression has an API, so your CI pipeline can open a window as a deploy step instead of someone remembering to click:

curl -s -X POST \
  "https://<controller-host>/controller/alerting/rest/v1/applications/<application-id>/action-suppressions" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "deploy-checkout-service",
    "disableAgentReporting": false,
    "suppressionScheduleType": "ONE_TIME",
    "timezone": "Australia/Sydney",
    "startTime": "2026-07-13T02:00:00",
    "endTime": "2026-07-13T02:30:00",
    "affects": { "affectedInfoType": "APPLICATION" }
  }'

While you're in the pipeline, record the deploy itself as an event on the Controller. It costs one call, shows up as a marker on every dashboard, and gives anyone investigating a violation the "what shipped" answer without leaving AppDynamics:

curl -s -X POST -H "Authorization: Bearer $TOKEN" \
  "https://<controller-host>/controller/rest/applications/<application-name>/events?severity=INFO&eventtype=APPLICATION_DEPLOYMENT&summary=checkout-service%202.14.0%20deployed"

Thirty minutes of suppression scoped to the service being deployed removes part one's violation storm without blinding you to everything else.

Layer 3: Policies and actions, the built-in reaction layer

A policy binds trigger events to actions. Go to Alert & Respond → Policies → Create Policy, select the triggering events (usually Health Rule Violation Started: Critical and Warning, plus Upgraded), and filter to specific health rules. A policy scoped to "any violation anywhere" recreates the noise you just removed. Then attach actions (policies docs).

Actions come in increasing strength.

Notifications. Email and SMS to people or groups. Fine as a floor. If it's your only action type, AppDynamics is a paging system with extra steps.

Diagnostic actions. The underused ones. Start a diagnostic session collects full transaction snapshots, with call graphs, for the affected business transactions over the next few minutes, so the evidence exists by the time a person looks. Take a thread dump captures a number of dumps at an interval from the affected nodes. Attach a diagnostic session to every latency-related critical policy. It costs nothing when nothing is wrong.

Remediation actions. Run a script or executable on problem nodes runs a script through the Machine Agent on the violating node. The script must already exist on that node under the Machine Agent's local-scripts directory:

# On each app node, alongside the Machine Agent
cat > /opt/appdynamics/machine-agent/local-scripts/recycle-app-pool.sh <<'EOF'
#!/bin/bash
systemctl restart checkout-service
echo "checkout-service restarted at $(date -u)" >> /var/log/appd-remediation.log
EOF
chmod +x /opt/appdynamics/machine-agent/local-scripts/recycle-app-pool.sh

In the action definition you reference the script path, set a timeout and, use this, check require approval before execution, which emails a designated approver a link to authorize the run. Script output is captured as an event on the Controller, so there's an audit trail (actions docs).

Remediation scripts are where DIY automation peaks and where it starts to rot. The script runs on the node, maintained by whoever wrote it, versioned nowhere, tested when it was written. Keep them to dumb, idempotent moves (restart a service, clear a temp directory) and manage them with your config management, not by hand on hosts.

Layer 4: HTTP request actions with templating

To push violations into Slack, PagerDuty, Jira or your own endpoint, create an HTTP Request Template (Alert & Respond → HTTP Request Templates), then an action that uses it. Templates support Velocity variables resolved from the triggering event (HTTP request templates docs):

{
  "summary": "${latestEvent.displayName}",
  "severity": "${latestEvent.severity}",
  "application": "${latestEvent.application.name}",
  "message": "${latestEvent.eventMessage}",
  "controller_link": "${latestEvent.deepLink}"
}

Set the method to POST, add auth headers, set the payload MIME type to application/json, and use the built-in Test button with a real past event before trusting it. The deepLink variable matters most in practice. It's the difference between a chat message someone reads and one someone clicks.

Layer 5: The REST APIs, pull instead of push

Everything above is push. For dashboards, dedup logic or your own tooling, poll the Controller with the token from the start of this guide.

Open health rule violations for an application over the last hour:

curl -s -H "Authorization: Bearer $TOKEN" \
  "https://<controller-host>/controller/rest/applications/<application-name>/problems/healthrule-violations?time-range-type=BEFORE_NOW&duration-in-mins=60&output=JSON"

Each violation includes the rule name, severity, affected entity, start time and a deep link. The events endpoint covers the policy lifecycle too:

curl -s -H "Authorization: Bearer $TOKEN" \
  "https://<controller-host>/controller/rest/applications/<application-name>/events?time-range-type=BEFORE_NOW&duration-in-mins=1440&event-types=POLICY_OPEN_CRITICAL,POLICY_OPEN_WARNING,POLICY_CLOSE_CRITICAL,POLICY_CLOSE_WARNING&severities=WARN,ERROR&output=JSON"

The evidence a diagnostic session collected is reachable the same way. Pull the slow and error snapshots for a business transaction in the violation window:

curl -s -H "Authorization: Bearer $TOKEN" \
  "https://<controller-host>/controller/rest/applications/<application-name>/request-snapshots?time-range-type=BEFORE_NOW&duration-in-mins=30&business-transaction-ids=<bt-id>&user-experience=SLOW,VERY_SLOW,ERROR&need-exit-calls=true&output=JSON" \
  | jq '.[] | {serverStartTime, timeTakenInMilliSecs, userExperience, summary}'

And the alerting configuration itself is exportable, which is useful for diffing and keeping your rules in version control:

curl -s -H "Authorization: Bearer $TOKEN" \
  "https://<controller-host>/controller/alerting/rest/v1/applications/<application-id>/health-rules" | jq .

Budget a day to do all five layers properly on one busy application. Done well, it cuts violation volume sharply and means every critical arrives with snapshots already collected.

---

What native AppDynamics covers, and what it leaves to you

The deeper difference is between collecting evidence and reading it. Every layer above makes the Controller quieter or faster to react, and diagnostic sessions mean the snapshots exist when you need them. Nothing in the stack opens those snapshots, finds the slow call and connects it to the deploy that introduced it.

The ceiling

Everything above shares one shape: predefined trigger, predefined step.

01Native

Predefined trigger

A policy matches a rule, a severity and an entity you chose in advance.

02Native

Predefined step

The same script, webhook or diagnostic session runs whatever the cause.

03Still manual

Read the evidence

Open the snapshots, find the slow call, match it to what changed, name the cause.

A policy can restart the service every time the error-rate rule fires. It restarts it whether the cause is a bad deploy, an exhausted connection pool or a downstream API returning 500s, because it can't tell which.

The diagnostic sessions you wired up collect snapshots with full call graphs, and then nobody reads them until an engineer opens the Controller, drills through snapshots, cross-references the release that went out 20 minutes earlier and forms a hypothesis. The native stack automates reaction. Investigation (reading the evidence, correlating it with what changed, naming a cause) is still a human, per-incident, 30-to-90-minute job. And every remediation script is a small liability with a shelf life.

What comes next

The investigation step is what part three of this series automates. CloudThinker connects to AppDynamics read-only, picks up each health rule violation, reads the snapshots and error details a person would, correlates them with recent releases and infrastructure state, and proposes a fix with evidence, with nothing changed until a named person approves it. Try CloudThinker free with 100 premium credits and no card required.