Zabbix Automation Starts With a Problem Queue You Can Trust

It's Monday, 9:04 a.m. Monitoring → Problems shows 312 entries. Somewhere in there are two real incidents: a replica that stopped replicating on Saturday and a disk that will fill by Thursday. They look exactly like the 310 entries that don't matter: the flapping CPU trigger on a batch host, the switch cascade from a five-minute maintenance reboot, the "High" severity alert that has been active for 41 days because everyone agreed it's fine.

This is the queue most Zabbix installations end up with, and it's why Zabbix automation efforts fail before they start. You can't automate responses to a signal you don't trust. Zabbix itself is rarely the problem. It's collecting data and evaluating triggers exactly as configured. The problem is that the configuration grew for five years and nobody ever took anything away.

This guide walks through the patterns that turn a Zabbix problem queue into noise: how each one develops, one concrete way to detect it in the frontend or with trigger expressions, and what it costs the on-call rotation.

This is part one of a three-part series. Part two is a hands-on guide to fixing these with native Zabbix features: trigger expressions, dependencies, actions and the Zabbix API. Part three covers putting an AI agent on top of the problem queue to triage and resolve continuously.

01Volume

Trigger sprawl

Stacked templates and inflated severity until High means nothing.

02Noise

Flapping

Instant thresholds with no recovery band, firing a dozen times an hour.

03Storms

Missing dependencies

One switch reboot, ninety problems, the real one on page three.

04Silence

Endless maintenance

Windows extended to a far-off date that work like a delete.

05Blind spots

Not supported items

Items that stopped collecting, and triggers that can no longer fire.

06Signal

What should page

Absence, trend, outside-in checks and windowed saturation.

---

Why the queue only ever grows

None of what follows is a Zabbix flaw. It comes from how configuration accumulates.

Templates add triggers nobody chose. Linking a template is one click and brings dozens of triggers. Unlinking or tuning them means knowing which ones matter, and nobody has time for that on the day a host goes live.

Thresholds are set once, against traffic that changes. A trigger tuned on launch day sits inside today's normal range a year later. It still technically works, so nobody revisits it.

Silencing is easier than fixing. A maintenance window or an ignored severity takes seconds. Fixing a trigger expression takes understanding. Under on-call pressure, the fast option wins every time.

Nothing subtracts. No process in a typical Zabbix install removes a trigger, closes a window or deletes an unsupported item. Every change is additive.

01Day 1

Clean queue

Templates linked, thresholds set against launch-day traffic.

02Month 3

First flaps

Traffic moves, thresholds don’t. Pages start resolving themselves.

03Year 1

Mutes pile up

Maintenance windows and ignored severities replace fixes.

04Year 5

312 problems

Two real incidents look identical to 310 that don’t matter.

---

1. Trigger sprawl and severity inflation

How it develops. Every host gets a template, every template ships dozens of triggers, and nobody prunes after linking. The official Linux agent template alone brings CPU, memory, swap, disk, inode, interface and service triggers. Multiply by 400 hosts and you're evaluating tens of thousands of triggers, most of which nobody consciously chose. Then severity inflates: a team wants their alert noticed, so it becomes "High". A year later half the queue is High or Disaster, and severity no longer sorts anything.

Detect it. Open Monitoring → Problems, clear all filters, and count by severity (toggling the severity filter checkboxes gives you counts per level). A healthy queue is a pyramid: many Information and Warning, few High, Disaster almost never. If yours is top-heavy, with more High than Warning, severity has stopped encoding urgency. Cross-check the raw trigger count per host under Data collection → Hosts (the Triggers column). Hosts with 80+ triggers from stacked templates are sprawl candidates.

The cost. Severity is the only field on-call uses to decide "phone down or phone up". When it's inflated, every decision means opening the event and reading history. Thirty seconds each, hundreds of times a week.

2. Flapping triggers that train people to ignore pages

How it develops. A trigger built on an instantaneous check fires the moment one value crosses the line and resolves the moment one value crosses back:

last(/Linux by Zabbix agent/system.cpu.util)>90

Any bursty workload turns this into a strobe light: PROBLEM, OK, PROBLEM, OK, twelve times an hour. There's no hysteresis and no time window, so the trigger flaps with the workload. After two weeks, on-call has learned the only lesson flapping teaches: this alert resolves itself, ignore it. That reflex doesn't stay limited to the flapping trigger. It spreads.

Detect it. Go to Reports → Top 100 triggers, set the range to the last 7 days, and sort by number of status changes. Anything firing more than a handful of times a day is either a real recurring problem (rare) or a flapping expression (usual). The fix is a windowed function with a separate recovery expression (trigger expression docs), which part two covers with exact syntax:

Problem:  avg(/Linux by Zabbix agent/system.cpu.util,5m)>90
Recovery: avg(/Linux by Zabbix agent/system.cpu.util,5m)<80

The cost. Flapping is the fastest way to destroy trust in a monitoring system. One flapping Disaster trigger that pages five times a night does more damage than fifty silent misconfigurations, because it turns paging from "wake up" into "swipe away".

3. The cascade nobody configured dependencies for

How it develops. A top-of-rack switch reboots for five minutes. Zabbix, correctly, reports every host behind it unreachable: forty hosts, forty "Zabbix agent is not available" problems, plus the ICMP triggers, plus every nodata trigger downstream. One root cause, ninety queue entries, and the switch trigger itself is buried on page three. Zabbix has first-class trigger dependencies to suppress exactly this (dependent problems stay hidden while the parent is down), but dependencies are configured per trigger, and in most installations they never were.

Detect it. You don't need a report. You need the memory of your last network blip. Open Monitoring → Problems, set the time filter to that window, and count the entries per root cause. More than three problems per actual failure means missing dependencies. To audit systematically, the Zabbix API exposes it directly: trigger.get with selectDependencies returns an empty array for every trigger that depends on nothing. Part two has the full curl version.

The cost. Cascades are what make people mute Zabbix in Slack. A channel that posts ninety messages for one switch reboot is a channel nobody reads during a real outage.

4. Maintenance windows that never end

How it develops. Someone creates a maintenance window for a migration: "no data collection, ends Sunday". The migration slips, so the window gets extended. Then extended again, this time to a date comfortably far away, and everyone forgets. The hosts are now unmonitored in production, indefinitely, and the queue looks cleaner for it, which is exactly why nobody notices. A maintenance window with a far-future end date is effectively a delete, except it doesn't show up in any diff.

Detect it. Open Data collection → Maintenance (Configuration → Maintenance on older versions) and read two columns: State and Active till. Expired windows that were never cleaned up are clutter. Active windows with end dates months out, or covering whole host groups, are the dangerous ones. Match each against a change ticket. The maintenance docs cover the "with data collection" mode, which suppresses alerts while still collecting. Most permanent mutes should have been that, or a targeted trigger fix, not a blackout.

The cost. This is the opposite failure to the others: instead of noise, silence. The incident that finally hits a permanently muted host group arrives with zero warning and no recent data to investigate with.

5. Items rotting in "Not supported"

How it develops. An item goes Not supported when Zabbix can't collect it: the key doesn't exist after an OS upgrade, a permission changed, a UserParameter script was removed, a filesystem was renamed. The item stops collecting, every trigger built on it stops evaluating, and Zabbix keeps running happily. Unlike a down host, an unsupported item raises no problem by default. It just rots, and each one is a trigger that can no longer fire.

Detect it. Go to Data collection → Hosts, click Items, and set the State filter to Not supported. Read the error column. It tells you exactly why collection fails. Check stuck checks too in Administration → Queue (queue docs), where items waiting longer than 10 minutes point to collection problems. For a permanent guardrail, Zabbix monitors itself: an internal item with key zabbix[items_unsupported] plus a trigger on its growth catches rot as it happens.

The cost. Every unsupported item is a blind spot you're confident you don't have. The disk-full trigger you rely on may be built on an item that stopped collecting in March.

6. The trigger patterns that actually carry signal

After pruning, what deserves to page? In practice a short list of patterns carries nearly all the real signal:

• Absence, not threshold. nodata(/host/agent.ping,5m)=1: the host stopped talking. Hard to inflate, hard to flap.
• Trend, not snapshot. Forecast-based disk triggers using timeleft(). "Full in 48 hours" is actionable. "82% used" is trivia.
• Service checks from outside. net.tcp.service and web scenarios that test what users experience, not what a process table says.
• Windowed saturation. Sustained avg() over 5 to 15 minutes on CPU, memory and connection pools, with recovery hysteresis.

What's not on the list: instant thresholds on utilization metrics, which generate most of the flaps, and per-host triggers duplicated for conditions a dependency should collapse.

---

One entry, read properly

Take the "High" problem from the opening, active for 41 days: /var: Disk space is critically low (used > 90%) on app-prod-07. Everyone agreed it's fine. There are three readings, and each predicts something different.

• A real capacity trend. Usage is still climbing, slowly. The item history shows a steady slope, and at some date it hits 100%.
• A stable plateau. Usage jumped once (a log retention change, a new cache) and has sat at 91% since. The threshold is wrong for this host, and the problem should be retuned, not ignored.
• A stale item. The item went Not supported weeks ago and the problem is frozen on its last value. Nothing about the disk is known at all.

The check is the item itself. In the frontend, open Monitoring → Latest data for the host and look at the item's state and its 30-day graph. Over the API, the same answer is one call:

{
  "jsonrpc": "2.0",
  "method": "item.get",
  "params": {
    "hostids": ["10584"],
    "search": { "key_": "vfs.fs.size[/var,pused]" },
    "output": ["itemid", "state", "error", "lastvalue", "lastclock"]
  },
  "id": 1
}

state: 1 with an error is reading three. state: 0 with a recent lastclock sends you to the graph: a slope is reading one, a flat line is reading two. In this case it was a flat line at 91% since a log retention change in August. The fix is a timeleft() trigger for this host and closing the problem, not another 41 days of scrolling past it.

That's one entry. The queue has 311 more, and each one needs the same few minutes of reading before anyone can say whether it matters.

---

The on-call math

Put numbers on a 300-entry queue. If triage takes 30 seconds per entry and the queue turns over twice a week, that's five engineer-hours a week spent telling noise from signal, before anyone fixes anything. The worse cost is about probability: every extra noise entry lowers the chance the two real problems get opened at all. Alert fatigue isn't a morale complaint. It's a detection-latency problem. The replica that broke Saturday gets found Monday not because Zabbix was slow, but because the queue hid it for 36 hours.

Cleanup is doable with native features. Part two walks through the exact trigger expressions, dependencies and Zabbix API calls. The catch is that hygiene decays: every new template link, host and "temporary" mute restarts the clock.

---

Keep the queue honest continuously

The audits above are point-in-time. The rot is continuous. CloudThinker connects to Zabbix read-only over the API and works the live problem queue the way a careful engineer would: separating outages from flaps from noise, flagging stale mutes and unsupported items, and proposing fixes. Changes go through Auto Mode, so you decide what runs on its own and what waits for approval. For a full walkthrough on a real estate, see CloudThinker × Zabbix: the full lifecycle.

A first question to try:

"List every Zabbix problem that has been open for more than 14 days. For each, tell me whether the underlying item is still collecting and whether the value is trending or flat. Propose only, change nothing."

Try CloudThinker free with 100 premium credits and no card required, or continue with the native-tools guide.