Zabbix Automation with Native Tools: Triggers, Actions, and the API

One line of configuration ends a flapping trigger for good:

Recovery expression: avg(/web01/system.cpu.util,5m)<70

That's it. The trigger that fired at 85% CPU now refuses to clear until the five-minute average drops below 70, and the fire-clear-fire-clear cycle that paged your on-call nine times last Tuesday is gone. The fix isn't hard. The problem is the other four hundred triggers on your server that nobody has audited to add it.

Part one of this series mapped the problem-queue reality (trigger sprawl, flapping, missing dependencies, permanent maintenance mutes) and how to detect each. This part is the fix session: tightening triggers, collapsing cascades, wiring escalations and scripting hygiene audits with nothing but what ships in Zabbix. Native features only, exact expressions and API calls, grouped by feature the way you'd actually work through them. Part three covers what happens when this runs continuously instead of once.

Frontend paths below use the Zabbix 6.4+ menu layout. On 6.0 LTS, swap Data collection for Configuration.

01Feature 1

Expressions

Recovery bands and time windows end most flapping.

02Feature 2

Dependencies

One upstream problem instead of forty shadows.

03Feature 3

Actions

An escalation ladder, plus remote commands for safe repeats.

04Feature 4

Maintenance

Planned silence with an end date and data still collected.

05Feature 5

API

Hygiene audits scripted across every trigger and window.

---

Feature 1: Trigger expressions, hysteresis and time windows

Two expression patterns eliminate most flapping. Both live in the trigger editor: Data collection → Hosts → Triggers → your trigger (expression docs).

Recovery expressions (hysteresis)

By default a trigger recovers the instant its problem expression goes false, so a metric oscillating around the threshold produces an event storm. Set OK event generation to Recovery expression and give the trigger a deadband:

Problem expression:  avg(/web01/system.cpu.util,5m)>85
Recovery expression: avg(/web01/system.cpu.util,5m)<70

The trigger fires above 85 and won't clear until the average falls below 70. The 15-point gap is your hysteresis band. Size it to the metric's normal jitter. Disk space, connection counts and queue depths all deserve the same treatment.

Time windows instead of single samples

last() evaluates one sample, so one collection blip pages a person:

last(/web01/net.if.in["eth0"])>100M        # fires on a single sample
min(/web01/net.if.in["eth0"],10m)>100M     # every sample high for 10 minutes
avg(/web01/system.cpu.util,5m)>85          # sustained average, absorbs spikes

The rule of thumb: min() over a window for "this must be persistently bad", avg() for "this is bad on balance", and last() only for discrete states (service down, file missing) where a single sample really is the truth. While you're in the editor, set the severity honestly. Part one covered why a queue full of "High" that nobody treats as high is worse than useless.

Feature 2: Trigger dependencies, collapse the cascade

When a rack switch dies, every host behind it goes unreachable and Zabbix dutifully opens forty problems. Trigger dependencies fix this in the data model: a dependent trigger stays quiet while its master trigger is in PROBLEM state, so the switch pages once and the forty shadows never do.

Setup is per trigger: open the dependent trigger, Dependencies tab, Add, and pick the master, typically the icmpping unreachable trigger on the upstream switch, router or hypervisor. Do it once on the template and every host linked to it inherits the dependency.

A practical first pass:

1. List your network chokepoints: uplink switches, firewalls, VPN concentrators, hypervisors.
2. For each, find its "unreachable" trigger (usually icmpping(...)=0 or nodata() on agent availability).
3. Make host-level triggers on everything downstream depend on it.

Dependencies chain (A depends on B depends on C), and a trigger can have several masters. Two failure modes to avoid: circular dependencies, which Zabbix rejects at save time, and over-broad masters that suppress real problems. Depend on the network path, not on "any problem on the DB server".

Feature 3: Actions and escalations, the response ladder

Triggers decide that something is wrong. Actions decide what happens next. Alerts → Actions → Trigger actions → Create action.

An action has conditions (which problems it matches) and operations (what it does, in escalating steps). A sane default ladder for severity High and above:

• Step 1 (immediately): message the on-call user group through your Slack, PagerDuty or email media type.
• Step 2 (after 15 minutes, unacknowledged): repeat to on-call plus the team lead.
• Step 3 (after 45 minutes): message the wider engineering channel.

Set the step duration on each operation, and add recovery operations so the same channels learn when the problem clears. An alert that never says "resolved" trains people to stop reading alerts. Escalations stop on acknowledgment only if you condition them on it, which is exactly the pressure valve you want: acknowledging in Monitoring → Problems halts the ladder.

Remote commands, the sharp edge

Operations can also run remote commands: restart a service, clear a temp directory, bounce a worker. Define the script under Alerts → Scripts with scope "Action operation", choose where it runs (Zabbix agent, server or proxy), and reference it from the operation. Running on the agent needs the agent config to allow it with AllowKey=system.run[*]. Scope it tighter than * in production.

Use them, but know what they are: an action executes what a trigger already decided. If the trigger is wrong (a flap, a threshold set two years ago, a symptom of a different root cause), the remote command fires anyway. The classic failure is the restart loop. A service OOMs, the trigger fires, the command restarts it, memory fills again, and it repeats every 20 minutes for a week while the leak ships in every release. The restart hides the signal that would have gotten the leak fixed. Keep remote commands for actions that are safe to repeat and cheap when wrong, and let the escalation ladder handle anything that needs judgment.

Feature 4: Maintenance windows that actually expire

Deploy windows and patch nights belong in maintenance windows, not in disabled triggers someone will forget to re-enable. Data collection → Maintenance → Create maintenance period.

Two disciplines make the feature safe:

1. Always set "Active till". A maintenance window with a far-future end date is a permanent mute dressed up as process. Part one showed how to find the ones you already have.
2. Prefer "With data collection". History keeps flowing, so when the window ends you have data and post-incident review still works. "No data collection" leaves a hole in every graph.

One-time periods handle deploys. Recurring daily, weekly or monthly periods handle backup windows and batch jobs that legitimately trip thresholds every night at 2 a.m. A recurring 90-minute window is strictly better than a threshold loosened around the clock to accommodate 90 minutes of load.

Feature 5: The Zabbix API, hygiene at scale

Everything above is per-trigger clicking. At four hundred triggers, you script the audit with the Zabbix API: JSON-RPC over HTTP, one endpoint. Create a token under Users → API tokens (a read-only user is enough for audits), then:

export ZABBIX_URL="https://zabbix.example.com"
export ZABBIX_TOKEN="your-api-token"

Snapshot the live problem queue with problem.get, Warning and above, newest first:

curl -s -X POST "$ZABBIX_URL/api_jsonrpc.php" \
  -H 'Content-Type: application/json-rpc' \
  -H "Authorization: Bearer $ZABBIX_TOKEN" \
  -d '{
    "jsonrpc": "2.0",
    "method": "problem.get",
    "params": {
      "output": ["eventid", "name", "severity", "clock", "acknowledged"],
      "severities": [2, 3, 4, 5],
      "sortfield": ["eventid"],
      "sortorder": "DESC"
    },
    "id": 1
  }' | jq '.result | length'

Run it weekly and chart the number. A queue that only grows is a queue nobody trusts.

Find triggers with no hysteresis. recovery_mode 0 means the trigger clears the instant the problem expression goes false, so every one of these is a flap candidate:

curl -s -X POST "$ZABBIX_URL/api_jsonrpc.php" \
  -H 'Content-Type: application/json-rpc' \
  -H "Authorization: Bearer $ZABBIX_TOKEN" \
  -d '{
    "jsonrpc": "2.0",
    "method": "trigger.get",
    "params": {
      "output": ["triggerid", "description", "expression"],
      "selectHosts": ["host"],
      "monitored": true,
      "filter": {"recovery_mode": 0}
    },
    "id": 2
  }' | jq '.result | length'

Build a flap leaderboard. Count problem events per trigger over the last week and sort:

SINCE=$(date -d '7 days ago' +%s)   # macOS: date -v-7d +%s
curl -s -X POST "$ZABBIX_URL/api_jsonrpc.php" \
  -H 'Content-Type: application/json-rpc' \
  -H "Authorization: Bearer $ZABBIX_TOKEN" \
  -d "{
    \"jsonrpc\": \"2.0\",
    \"method\": \"event.get\",
    \"params\": {
      \"output\": [\"objectid\"],
      \"source\": 0, \"object\": 0, \"value\": 1,
      \"time_from\": $SINCE
    },
    \"id\": 3
  }" | jq -r '.result[].objectid' | sort | uniq -c | sort -rn | head

Any trigger ID that appears dozens of times in a week goes to the top of your hysteresis worklist. Feed the IDs back into trigger.get to resolve names. The same pattern extends naturally: maintenance.get to list windows and their active_till, trigger.get with selectDependencies to find triggers with no dependency wiring. (Bearer-token auth needs Zabbix 6.4+. On older versions, pass the token in the request body's auth field instead.)

---

One storm through the native stack

It's 02:14 and 38 "Zabbix agent is not available" problems open within the same minute, all on hosts in the app-prod group. Your action sends 38 messages to Slack, because nobody wired dependencies for this rack. Before anyone starts restarting things, there are three readings.

01Reading 1

Network path

A switch or uplink died. Predicts one upstream ICMP problem and hosts behind it.

02Reading 2

Hypervisor

One host under many VMs failed. Predicts every affected VM shares it.

03Reading 3

Agents only

The hosts are fine, the agents died. Predicts ping and outside checks still pass.

Each one has a check.

The network path shows up as an upstream ICMP problem that opened first. Pull everything that opened in the window, oldest first, and see what leads:

curl -s -X POST "$ZABBIX_URL/api_jsonrpc.php" \
  -H 'Content-Type: application/json-rpc' \
  -H "Authorization: Bearer $ZABBIX_TOKEN" \
  -d '{
    "jsonrpc": "2.0",
    "method": "problem.get",
    "params": {
      "output": ["eventid", "name", "clock", "objectid"],
      "time_from": 1759889400,
      "sortfield": ["eventid"],
      "sortorder": "ASC"
    },
    "id": 4
  }' | jq -r '.result[] | [.clock, .name] | @tsv' | head

A failed hypervisor shows up as every affected host being a VM on the same physical host, with that host's own problems (or silence) in the same minute. If you monitor vSphere with the VMware template, the hypervisor is a host in Zabbix too, and its problem will be in the list above.

Dead agents on live hosts show up as ICMP and outside service checks still passing while the agent items fail. Check the ping item on the affected hosts:

curl -s -X POST "$ZABBIX_URL/api_jsonrpc.php" \
  -H 'Content-Type: application/json-rpc' \
  -H "Authorization: Bearer $ZABBIX_TOKEN" \
  -d '{
    "jsonrpc": "2.0",
    "method": "item.get",
    "params": {
      "groupids": ["27"],
      "filter": {"key_": "icmpping"},
      "output": ["hostid", "lastvalue", "lastclock"]
    },
    "id": 5
  }' | jq -r '.result[].lastvalue' | sort | uniq -c

Tonight problem.get showed no switch or hypervisor problem at all, and icmpping was 1 on all 38 hosts. The web scenarios for the apps on them were green. The hosts were fine. A configuration management run at 02:13 had pushed a zabbix_agent2.conf with a typo, and every agent failed to come back after the restart. Reading three. The fix was a revert in the config repo, and an engineer who'd started by paging the network team would have spent twenty minutes on the wrong problem.

Look at what the native stack did here. The triggers fired correctly. The action delivered 38 messages correctly. Nothing in the stack connected the storm to the config push a minute earlier, because Zabbix doesn't know your config management exists. The reasoning (which reading, which check first) was a person reading timestamps at 2 a.m.

---

What each native feature solves, and what it leaves

01Triggers

Decide

An expression crossed a line, so a problem opens.

02Actions

Execute

Messages, escalations and remote commands run on what the trigger decided.

03Still manual

Question

Whether the trigger was right: outage, flap, capacity trend or noise.

---

The ceiling of native automation

Do all of this and your Zabbix install will be much quieter. The work often cuts alert volume by half or more. But look at the shape of what you built. Recovery expressions, dependencies, escalations, remote commands: every one of them executes a decision a trigger already made. Nothing in the stack asks whether the trigger was right, whether this CPU alert is an outage, a flap, a capacity trend or noise from a threshold nobody has revisited since 2023. That investigation is still a person opening dashboards at 3 a.m. And the audit you just scripted is a point-in-time pass. New hosts, templates and triggers start rebuilding the mess the day after you finish.

The approval question you'll meet again

Remote commands already forced one decision on you: which fixes are safe to run unattended. In part three, writes like a trigger edit, a maintenance window or a host action go through Auto Mode, so you make the same call: what runs on its own and what waits for approval. If you've already decided which remote commands are safe, you've done half the work.

---

What comes next

The layer Zabbix doesn't ship is judgment: something that reads the live problem queue, checks each problem against host history and related triggers, and proposes the right fix (a trigger change, a scoped maintenance window or a host action) for a person to approve. That's what CloudThinker adds on top of the Zabbix API, read-only by default, and it's part three of this series. For the longer version on a real estate, see CloudThinker × Zabbix: the full lifecycle.

To try it against your own queue first:

"38 agent-unavailable problems opened at 02:14. Check for an upstream switch or hypervisor problem, check whether ping still answers, look for changes in the minute before, and tell me which explanation the evidence supports. Propose only, change nothing."

Try CloudThinker free with 100 premium credits and no card required.