Zabbix Automation with AI Agents: From Problem Queue to Resolution

There's a difference between muting a host at 3 a.m. and putting it in a maintenance window that carries its own evidence and its own expiry date. The first is a decision nobody will remember making. The second is an action someone can review at 9 a.m., see why it happened, and watch end on schedule. Most Zabbix automation stops at the first kind: remote commands and escalations that execute whatever a trigger already decided. This post is about the second kind, an action layer that investigates before it acts and leaves a trail either way.

This is part three of the Zabbix automation series. Part one mapped the problem-queue reality: trigger sprawl, flapping, missing dependencies, permanent maintenance windows, and the handful of trigger patterns that carry real signal. Part two covered the DIY toolkit: recovery expressions, trigger dependencies, actions and escalations, and hygiene audits over the Zabbix API with problem.get and trigger.get.

Part two ended at a ceiling worth restating. Zabbix actions execute what a trigger already decided. Nothing in the native stack asks whether the trigger was right: whether this problem is a real outage, a flap that has fired forty times this month, a capacity trend that needed attention last week, or noise that should never have paged anyone. That call still lands on whoever is on call. This part covers what happens when CloudThinker makes it first.

Officially listed on Zabbix. CloudThinker is a vendor-supported integration in the Zabbix integrations catalog. It uses the Zabbix API as Zabbix documents it: standard methods, standard authentication, no agent-side patches or forks. If you run Zabbix 6.0 LTS or later, the connection below works as described.

---

Why actions stop where investigation starts

A Zabbix action is a rule: these conditions, these operations, in this order. That's exactly right for routing, and part two built a good ladder. The trouble is that an action has no way to look at what happened last time.

Take a remote command that restarts a worker when its service check fails. It works. The service comes back, the problem resolves, the action log says success. If it happens nine times in a week, the action runs nine times, because each run is judged alone. Whether those nine restarts are a memory leak, a traffic surge or a trigger that's too tight is a question that branches on evidence: memory shape, queue depth, release timing. A person answers it by reading history. An action never reads history at all.

---

Connecting: a read-only API token, about five minutes

01Monitoring

Zabbix

• Problems and eventsproblem.get, event.get
• Triggers and itemstrigger.get, item.get, history.get
• Actionsaction.get, including remote commands
• Maintenancemaintenance.get, create when approved

Read-only API token

02Investigation

CloudThinker

• TriageOutage, flap, capacity or noise
• CorrelateEvent history, related items, changes
• ProposeTrigger fix, scoped window, host action
• VerifyResolved, and stays resolved

Nothing installed on hosts

03Where the team works

Chat and on-call

• Slack or TeamsFindings and approvals in the thread
• PagerDutyPages only on confirmed outages
• JiraRoot-cause tickets with evidence attached

Same channels, better input

The connection is deliberately boring. You create a dedicated Zabbix user with read permission on the host groups you want covered, generate an API token for it (Users → API tokens in the Zabbix 6.4+ frontend, Administration → General → API tokens on 6.0 LTS), and paste the token and your Zabbix URL into CloudThinker. No agent installs, no database access, no write permissions at connection time.

That read-only scope covers everything CloudThinker does by default: pulling the live problem queue, reading trigger configurations and expressions, walking event history per host and inspecting existing maintenance windows. The Zabbix connection guide lists the exact permission set, so your security review can see what's granted and what isn't. Write scope comes later, per host group, only if you decide to grant it.

---

What it does with your problem queue

Once connected, CloudThinker works the queue the way a careful senior engineer would, except continuously, and with the patience to check history every single time.

Triage: four buckets, not one wall of red

01Bucket 1

Outage

Down or degraded, confirmed by related items and reachability.

02Bucket 2

Flap

Cycled PROBLEM and OK repeatedly. Counted from history, not current state.

03Bucket 3

Capacity

Trending toward exhaustion. The number that matters is time to full.

04Bucket 4

Noise

A breach with nothing behind it, usually a trigger part one flagged.

This is the step native Zabbix skips. An escalation fires on severity. Triage fires on evidence.

Correlation: every problem gets its history read

For each problem, CloudThinker pulls the host's event history and the state of related triggers. A "disk 85% full" problem on a host that has hovered at 84 to 86% for six months is a different thing from the same trigger on a host that climbed ten points this week. A batch of forty unreachable-host alerts that all started in the same minute, all behind one switch, is one problem wearing forty costumes: the dependency chain you never configured, rebuilt after the fact.

Proposal: a specific fix, scoped and reversible

The output of an investigation isn't a summary. It's a proposed action, sized to what the evidence supports:

• Trigger change. A concrete revised expression, usually adding hysteresis with a recovery expression or replacing a last() check with an avg() or min() window, exactly the patterns from part two, with the flap history attached as justification.
• Scoped maintenance window with an expiry. For a host under known, bounded disruption: a maintenance.create proposal covering only the affected hosts, with a hard end time. Never the open-ended mute that becomes permanent.
• Host-level action. Acknowledging and closing stale problems with a reason attached, or flagging a host whose items have gone unsupported and are silently rotting.

---

One restart loop, end to end

01Signal

Service down

The trigger fires and a remote command restarts the worker.

02History

Ninth time

Event history shows the same problem nine times this week.

03Evidence

Investigate

Memory shape, load, release timing. Leak, load or flap.

04Output

Proposal

Keep the restart as a stopgap, fix the cause, change the action.

05Close

Verify

No restarts for the window you set after the fix ships.

Here's a case the remote command hid for a week. The trigger queue-worker is not running on worker-03 fires at 04:40. The action's remote command restarts the service, the problem resolves at 04:41, and nothing pages.

1. Read the history first. CloudThinker pulls the trigger's events for the last seven days with event.get and finds nine PROBLEM events, all resolved within two minutes by the same remote command. This isn't a one-off. The question is why it keeps dying.

2. Name the readings. A memory leak, which predicts memory climbing steadily between restarts until the process is killed. Load growth, which predicts memory tracking queue depth. Or a trigger that's too tight, which predicts the process was never actually down, just slow to answer a check during a normal restart.

3. Check memory shape. It pulls the worker's resident memory for the week:

{
  "jsonrpc": "2.0",
  "method": "history.get",
  "params": {
    "itemids": ["48211"],
    "history": 3,
    "time_from": 1759363200,
    "output": "extend",
    "sortfield": "clock",
    "sortorder": "ASC"
  },
  "id": 1
}

The series is a clean sawtooth. After each restart, proc.mem[queue-worker,,,,rss] climbs at a steady rate until it hits the service's memory limit, then the process dies. The host collects the kernel log with a log item, and it shows an OOM kill a few seconds before each problem. The trigger was right: the process really was down. The flap reading is out.

4. Check load. The queue depth item for the same week is flat. Memory climbs at the same rate at 4 a.m. as at peak. Load growth doesn't explain it.

5. Check for a change. The first of the nine events is Tuesday at 11:20. CloudThinker checks the change sources you've connected and finds release 2.14.0 of the worker deployed at 10:55 that day. Before Tuesday, the same memory item is flat for a month.

6. Write the finding. The thread gets one message: memory leak introduced in 2.14.0, with the sawtooth graph, the OOM lines, the flat queue depth and the release. Three proposals. Keep the remote command as a stopgap, because restarting really is safe here. Add a second trigger that fires when the service problem occurs more than three times in 24 hours, so the next loop pages someone instead of hiding. And open a Jira ticket for the worker team with the evidence, suggesting a rollback to 2.13.x until the leak is fixed.

The trigger creation and the ticket go through Auto Mode like any other write. Once the fix ships, CloudThinker watches the memory item and the trigger, and confirms the sawtooth is gone and no restart has fired for the window you set.

For a different walk through the same loop, a disk-latency regression traced to a release across two database hosts, see CloudThinker × Zabbix: the full lifecycle.

---

Write access is earned

CloudThinker acts within Auto Mode, so you decide what runs on its own and what waits for approval. Maintenance windows on confirmed flaps are low-stakes. Trigger rewrites deserve a closer look, because they change what Zabbix considers a problem.

Every action lands in an evidence trail: the problem, the history read, the classification, the proposal, who approved it, what changed in Zabbix and when. The 3 a.m. mute becomes a reviewable object instead of a mystery.

---

What a first pass typically finds

The numbers below are illustrative, a composite of what the first analysis tends to surface on a mid-market Zabbix installation of around 400 hosts with a busy problem queue. Yours will differ.

Look at the shape. Five of six rows are the hygiene debt from part one, and the sixth is the real signal that debt was burying. That ratio (most of the queue is self-inflicted, and the real problem is hiding inside it) is the normal first-pass result, not the exception.

---

Why this beats the traditional workflow

The deeper difference is what gets questioned. Native automation trusts the trigger and acts on it. CloudThinker treats the trigger as one piece of evidence, reads the rest, and only then proposes what to do. That's the part that used to happen at 3 a.m., or never.

---

Getting started

Three steps. None needs write access on day one.

1. Create a read-only API user and token on the host groups you want covered, following the Zabbix connection guide.

2. Run read-only triage. Nothing changes in Zabbix. Ask from chat:

"Triage all active Zabbix problems. Classify each as outage, flap, capacity or noise, and tell me which ones have fired more than ten times this month."

"Find every problem that a remote command resolved more than three times this week, and tell me whether the restarts are hiding a cause."

"Find every maintenance window older than 30 days. For each, tell me which hosts it covers and whether the original problem still exists. Propose replacements with expiry dates."

"Review the triggers that fired most in the last two weeks and draft improved expressions with hysteresis. Propose only, change nothing. Just show me the diffs."

3. Hand off routine fixes once the team trusts the triage.

"For the staging host group only, put confirmed flapping triggers in a 2-hour maintenance window. Trigger changes still come to me for review."

Each answer cites the underlying Zabbix data, the same problem.get, event.get and trigger.get evidence you pulled by hand in part two, so you can check it rather than trust it.

---

What it won't do

Because "AI with API access to monitoring" should raise exactly these questions:

• Read-only by default. The connection token grants no write scope. Changing a trigger, creating a maintenance window or acknowledging a problem needs you to grant write permission, and then goes through Auto Mode.
• No open-ended silences. Maintenance windows it proposes carry expiry times. It won't create the permanent mute part one warned about. Unwinding that pattern is one of the reasons it exists.
• No invisible actions. Every proposal and execution is in the evidence trail with its justification and approver. If it isn't in the trail, it didn't happen.
• No overriding your escalations. Existing Zabbix actions and on-call paths stay intact. CloudThinker reduces what reaches them. It doesn't intercept it.

---

Related reading

• CloudThinker × Zabbix: the full monitoring lifecycle
• Part one: a problem queue you can trust
• Part two: triggers, actions and the API
• Auto Mode: you decide what runs on its own

Closing the series

You know which Zabbix signals matter, and you know how far native trigger expressions and escalations go. What's left is the judgment step in the middle of the night: reading the history, questioning the trigger, and proposing a fix that addresses the cause. That step can now run continuously, with an evidence trail attached and a person approving every write until you decide otherwise.

Try CloudThinker free with 100 premium credits and no card required. Connect Zabbix read-only and see your own first-pass findings table within the hour.