Zabbix Automation with Native Tools: Triggers, Actions, and the API

One line of configuration ends a flapping trigger forever:

Recovery expression: avg(/web01/system.cpu.util,5m)<70

That's it. The trigger that fired at 85% CPU now refuses to clear until the five-minute average drops below 70, and the fire-clear-fire-clear cycle that paged your on-call nine times last Tuesday is gone. The problem is not that the fix is hard. The problem is the other four hundred triggers on your server that nobody has audited to add it.

This is part two of our three-part Zabbix automation series. Part one mapped the problem-queue reality — trigger sprawl, flapping, missing dependencies, permanent maintenance mutes — and how to detect each. This article is the fix session: tightening triggers, collapsing cascades, wiring escalations, and scripting hygiene audits with nothing but what ships in Zabbix. Native features only, exact expressions and API calls, grouped by feature the way you'd actually work through them. Part three covers what happens when this runs continuously instead of once.

Frontend paths below use the Zabbix 6.4+ menu layout; on 6.0 LTS, swap Data collection for Configuration.

Feature 1: Trigger expressions — hysteresis and time windows

Two expression patterns eliminate most flapping. Both live in the trigger editor: Data collection → Hosts → Triggers → your trigger (expression docs).

Recovery expressions (hysteresis)

By default a trigger recovers the instant its problem expression goes false — so a metric oscillating around the threshold generates an event storm. Set OK event generation to Recovery expression and give the trigger a deadband:

Problem expression:  avg(/web01/system.cpu.util,5m)>85
Recovery expression: avg(/web01/system.cpu.util,5m)<70

The trigger fires above 85 and will not clear until the average falls below 70. The 15-point gap is your hysteresis band; size it to the metric's normal jitter. Disk space, connection counts, and queue depths all deserve the same treatment.

Time windows instead of single samples

last() evaluates one sample, so one scrape blip pages a human:

last(/web01/net.if.in["eth0"])>100M        # fires on a single sample
min(/web01/net.if.in["eth0"],10m)>100M     # every sample high for 10 minutes
avg(/web01/system.cpu.util,5m)>85          # sustained average, absorbs spikes

The rule of thumb: min() over a window for "this must be persistently bad," avg() for "this is bad on balance," and last() only for discrete states (service down, file missing) where a single sample genuinely is the truth. While you're in the editor, check the severity honestly — part one covered why a queue full of "High" that nobody treats as high is worse than useless.

Feature 2: Trigger dependencies — collapse the cascade

When a rack switch dies, every host behind it goes unreachable and Zabbix dutifully opens forty problems. Trigger dependencies fix this at the data model level: a dependent trigger stays quiet while its master trigger is in PROBLEM state, so the switch pages once and the forty shadows never do.

Setup is per trigger: open the dependent trigger, Dependencies tab, Add, pick the master — typically the icmpping unreachable trigger on the upstream switch, router, or hypervisor. Do it once on the template and every host linked to the template inherits it.

The practical sequence for a first pass:

1. List your network chokepoints — uplink switches, firewalls, VPN concentrators, hypervisors.
2. For each, find its "unreachable" trigger (usually icmpping(...)=0 or nodata() on agent availability).
3. Make host-level triggers on everything downstream depend on it.

Dependencies chain (A depends on B depends on C) and a trigger can have several masters. The failure mode to avoid: circular dependencies, which Zabbix rejects at save time, and over-broad masters that suppress real problems — depend on the network path, not on "any problem on the DB server."

Feature 3: Actions and escalations — the response ladder

Triggers decide that something is wrong; actions decide what happens next. Alerts → Actions → Trigger actions → Create action.

An action has conditions (which problems it matches) and operations (what it does, in escalating steps). A sane default ladder for severity High and above:

• Step 1 (immediately): message to the on-call user group via your Slack/PagerDuty/email media type.
• Step 2 (after 15 minutes, unacknowledged): repeat to on-call plus the team lead.
• Step 3 (after 45 minutes): message the wider engineering channel.

Set the step duration on each operation, and add recovery operations so the same channels learn when the problem clears — an alert that never says "resolved" trains people to stop reading alerts. Escalations only proceed while the problem is unacknowledged if you condition them on acknowledgment, which is exactly the pressure valve you want: acknowledging in Monitoring → Problems halts the ladder.

Remote commands — the sharp edge

Operations can also run remote commands: restart a service, clear a temp directory, bounce a worker. Define the script under Alerts → Scripts with scope "Action operation," choose where it executes (Zabbix agent, server, or proxy), and reference it from the operation. Executing on the agent requires the agent config to allow it — AllowKey=system.run[*] (scope it tighter than * in production).

Use them, but understand what they are: an action executes what a trigger already decided. If the trigger is wrong — a flap, a threshold set two years ago, a symptom of a different root cause — the remote command fires anyway. The classic failure is the restart loop: service OOMs, trigger fires, command restarts it, memory fills again, repeat every 20 minutes for a week while the underlying leak ships in every release. The restart masks the signal that would have gotten it fixed. Reserve remote commands for actions that are safe under repetition and cheap when wrong, and let the escalation ladder handle everything that needs judgment.

Feature 4: Maintenance windows that actually expire

Deploy windows and patch nights belong in maintenance windows, not in disabled triggers someone will forget to re-enable. Data collection → Maintenance → Create maintenance period.

Two disciplines make the feature safe:

1. Always set "Active till." A maintenance window with a far-future end date is a permanent mute wearing a process costume — part one showed how to find the ones you already have.
2. Prefer "With data collection." History keeps flowing, so when the window ends you have data, and post-incident review still works. "No data collection" leaves a blind spot in every graph.

One-time periods handle deploys; recurring daily/weekly/monthly periods handle backup windows and batch jobs that legitimately trip thresholds every night at 2 a.m. — a recurring 90-minute window is strictly better than a threshold loosened 24/7 to accommodate 90 minutes of load.

Feature 5: The Zabbix API — hygiene at scale

Everything above is per-trigger clicking. At four hundred triggers, you script the audit with the Zabbix API — JSON-RPC over HTTP, one endpoint. Create a token under Users → API tokens (a read-only user is enough for audits), then:

export ZABBIX_URL="https://zabbix.example.com"
export ZABBIX_TOKEN="your-api-token"

Snapshot the live problem queue with problem.get — Warning and above, newest first:

curl -s -X POST "$ZABBIX_URL/api_jsonrpc.php" \
  -H 'Content-Type: application/json-rpc' \
  -H "Authorization: Bearer $ZABBIX_TOKEN" \
  -d '{
    "jsonrpc": "2.0",
    "method": "problem.get",
    "params": {
      "output": ["eventid", "name", "severity", "clock", "acknowledged"],
      "severities": [2, 3, 4, 5],
      "sortfield": ["eventid"],
      "sortorder": "DESC"
    },
    "id": 1
  }' | jq '.result | length'

Run it weekly and chart the number. A queue that only grows is a queue nobody trusts.

Find triggers with no hysteresis — recovery_mode 0 means the trigger clears the instant the problem expression goes false, i.e., every one of these is a flap candidate:

curl -s -X POST "$ZABBIX_URL/api_jsonrpc.php" \
  -H 'Content-Type: application/json-rpc' \
  -H "Authorization: Bearer $ZABBIX_TOKEN" \
  -d '{
    "jsonrpc": "2.0",
    "method": "trigger.get",
    "params": {
      "output": ["triggerid", "description", "expression"],
      "selectHosts": ["host"],
      "monitored": true,
      "filter": {"recovery_mode": 0}
    },
    "id": 2
  }' | jq '.result | length'

Build a flap leaderboard — count problem events per trigger over the last week and sort:

SINCE=$(date -d '7 days ago' +%s)   # macOS: date -v-7d +%s
curl -s -X POST "$ZABBIX_URL/api_jsonrpc.php" \
  -H 'Content-Type: application/json-rpc' \
  -H "Authorization: Bearer $ZABBIX_TOKEN" \
  -d "{
    \"jsonrpc\": \"2.0\",
    \"method\": \"event.get\",
    \"params\": {
      \"output\": [\"objectid\"],
      \"source\": 0, \"object\": 0, \"value\": 1,
      \"time_from\": $SINCE
    },
    \"id\": 3
  }" | jq -r '.result[].objectid' | sort | uniq -c | sort -rn | head

Any trigger ID appearing dozens of times in a week goes to the top of your hysteresis worklist; feed the IDs back into trigger.get to resolve names. The same pattern extends naturally — maintenance.get to list windows and their active_till, trigger.get with selectDependencies to find hosts with no dependency wiring. (Bearer-token auth requires Zabbix 6.4+; on older versions pass the token in the request body's auth field instead.)

The ceiling of native automation

Do all of this and your Zabbix install will be dramatically quieter — the work typically cuts alert volume by half or more. But notice the shape of what you built. Recovery expressions, dependencies, escalations, remote commands: every one of them executes a decision a trigger already made. Nothing in the stack asks whether the trigger was right — whether this CPU alert is an outage, a flap, a capacity trend, or noise from a threshold nobody has revisited since 2023. That investigation is still a human opening dashboards at 3 a.m. And the audit you just scripted is a point-in-time pass: new hosts, new templates, and new triggers start regenerating the mess the day after you finish.

What comes next

The layer Zabbix doesn't ship is judgment: something that reads the live problem queue, correlates each problem against host history and related triggers, and proposes the right fix — trigger tweak, scoped maintenance window, or host-level action — with a human approving. That's what CloudThinker agents add on top of the Zabbix API, read-only by default, and it's part three of this series. Or try CloudThinker free — 100 premium credits, no card required.