Zabbix Problem Triage
Overview
A Zabbix problem is an open problem event: a trigger (or an internal or
service check) went into PROBLEM state and has not recovered. Triage means
deciding which problems need a human now, which customer they belong to, and
whether someone has already been told. Everything here is read-only;
acknowledging, changing severity or closing happens in the Zabbix frontend.
Read api-patterns first for headers, the shared
get parameters, and event-ID paging. No tool in this workflow is
admin-gated in Conduit.
Anti-triggers
- Hosts with no monitoring, or a site proxy that went silent - problems
only exist for things being monitored; use
monitoring-coverage.
- Monthly availability numbers for a customer - use
sla-reporting.
Key Concepts
| Field (problem) |
Meaning |
eventid |
Problem event ID; also the paging key |
objectid |
The trigger ID when object is 0 (the default) |
name |
Trigger name with macros expanded at the time of the event |
severity |
0 not classified, 1 information, 2 warning, 3 average, 4 high, 5 disaster |
clock |
Unix time the problem started |
acknowledged |
1 when someone acknowledged it in Zabbix |
suppressed |
1 when maintenance or a manual suppression hides it |
r_eventid |
Recovery event ID; non-zero means it already resolved |
cause_eventid |
Non-zero on a symptom problem; points at its cause |
zabbix_problems_list returns no host. Resolve the host through the
trigger (zabbix_triggers_list with triggerids set to the problems'
objectid values and selectHosts) or through the event
(zabbix_events_list with eventids and selectHosts).
- MSPs usually keep one host group per customer or site. Get the group IDs
from
zabbix_hostgroups_list and pass groupids to scope everything else.
Some instances tag hosts by customer instead; then use tags.
Common Workflows
Open-problem sweep (one customer or all)
zabbix_hostgroups_list with search: {"name": "<customer>"} and
with_monitored_hosts: true to get the customer's group IDs. Skip for an
all-customer sweep.
zabbix_problems_list with groupids, severities: [3,4,5],
sortfield: "eventid", sortorder: "DESC", selectTags: "extend",
selectSuppressionData: "extend", limit: 200. Add
symptom: false to see only cause problems first.
- Collect the
objectid values and call zabbix_triggers_list once with
triggerids, selectHosts: ["host","name"],
selectHostGroups: ["name"], expandDescription: true and
output: ["triggerid","description","priority","comments","url"]. This maps
every problem to its host and customer group in one call, and the trigger
comments often carry the runbook note.
- If any problem came back on a host in maintenance or with
suppressed: 1, confirm with zabbix_maintenance_list (hostids,
selectTimeperiods: "extend") before calling it noise.
- Rank: unacknowledged disaster/high first, then by age (
clock), then
problems shared by many hosts in one group (a site-wide outage beats ten
single-host warnings).
Was anyone paged?
zabbix_alerts_list with eventids set to the problem event IDs and
selectHosts: ["name"]. Each alert shows sendto, status
(0 not sent yet, 1 sent, 2 failed) and error. A disaster with no alert, or
with failed alerts, is an escalation gap to report, not just a problem.
Is it flapping?
zabbix_events_list with objectids: ["<triggerid>"],
time_from 24-72 hours back, sortfield: "clock", sortorder: "DESC",
output: ["eventid","clock","value","severity"]. Many 1/0 (problem/OK)
pairs in a short window means a flapping trigger: the fix is a threshold or
hysteresis change on the trigger, not a ticket per occurrence.
Currently-firing triggers (alternative view)
zabbix_triggers_list with only_true: true, monitored: true,
skipDependent: true, expandDescription: true, filter: {"value": 1},
min_severity: 3, selectHosts: ["name"], selectLastEvent: "extend".
skipDependent hides triggers whose parent (for example the site uplink)
is already in problem, which is usually what a NOC wants.
Gotchas
- Suppressed is not resolved. Problems on hosts in maintenance are
hidden in the frontend's default views but still returned unless you pass
suppressed: false. Say which you included.
recent: true mixes in resolved problems. Check r_eventid before
calling something open.
- Symptom problems inflate counts. With cause and symptom linking,
one router outage can own dozens of symptom problems. Count causes.
- Severity is per problem, not per trigger. Someone may have changed the
problem's severity in the frontend; the trigger's
priority keeps the
original.
acknowledged is only Zabbix's view. The ticket may live in the PSA
without an acknowledgement in Zabbix. Do not report "nobody is working
this" from acknowledged: 0 alone.
- Item values are not available.
zabbix_items_list and
zabbix_history_get are opt-in and not listed in Conduit. The trigger
name (with expanded macros) and zabbix_trends_get are what you have.
Output
Group by customer (host group), then severity. Per problem: host, problem
name, severity, age, acknowledged yes/no, suppressed yes/no, alert sent /
failed / none, and a one-line next step. End with counts: open causes,
symptoms, suppressed, and any paging gaps.
Related Skills
1---2name: zabbix-problem-triage3description: >4 NOC triage of Zabbix problems for MSPs: pulling open problems by severity5 and customer host group, resolving each problem to its host through the6 trigger or event, separating cause from symptom and suppressed problems,7 checking maintenance and acknowledgement state, confirming whether an8 alert was actually sent, and reading recent event history for flapping.9 Read-only: the plugin cannot acknowledge or close problems.10when_to_use: >-11 When reviewing what is currently broken in a customer's Zabbix, ranking12 open problems, or investigating one problem before opening or updating a13 ticket. Use when: zabbix problems, open problems, zabbix alerts today,14 what's down, NOC triage, zabbix disaster, unacknowledged problems, was15 anyone paged, flapping trigger, or zabbix shift handover.16metadata:17 source: https://skillmd.com/skills/wyre-ai/zabbix-problem-triage18---1920# Zabbix Problem Triage2122## Overview2324A Zabbix **problem** is an open problem event: a trigger (or an internal or25service check) went into PROBLEM state and has not recovered. Triage means26deciding which problems need a human now, which customer they belong to, and27whether someone has already been told. Everything here is read-only;28acknowledging, changing severity or closing happens in the Zabbix frontend.2930Read [api-patterns](../api-patterns/SKILL.md) first for headers, the shared31get parameters, and event-ID paging. No tool in this workflow is32admin-gated in Conduit.3334## Anti-triggers3536- **Hosts with no monitoring, or a site proxy that went silent** - problems37 only exist for things being monitored; use38 [monitoring-coverage](../monitoring-coverage/SKILL.md).39- **Monthly availability numbers for a customer** - use40 [sla-reporting](../sla-reporting/SKILL.md).4142## Key Concepts4344| Field (problem) | Meaning |45|-----------------|---------|46| `eventid` | Problem event ID; also the paging key |47| `objectid` | The trigger ID when `object` is 0 (the default) |48| `name` | Trigger name with macros expanded at the time of the event |49| `severity` | 0 not classified, 1 information, 2 warning, 3 average, 4 high, 5 disaster |50| `clock` | Unix time the problem started |51| `acknowledged` | 1 when someone acknowledged it in Zabbix |52| `suppressed` | 1 when maintenance or a manual suppression hides it |53| `r_eventid` | Recovery event ID; non-zero means it already resolved |54| `cause_eventid` | Non-zero on a **symptom** problem; points at its cause |5556- `zabbix_problems_list` returns **no host**. Resolve the host through the57 trigger (`zabbix_triggers_list` with `triggerids` set to the problems'58 `objectid` values and `selectHosts`) or through the event59 (`zabbix_events_list` with `eventids` and `selectHosts`).60- MSPs usually keep one host group per customer or site. Get the group IDs61 from `zabbix_hostgroups_list` and pass `groupids` to scope everything else.62 Some instances tag hosts by customer instead; then use `tags`.6364## Common Workflows6566### Open-problem sweep (one customer or all)67681. `zabbix_hostgroups_list` with `search: {"name": "<customer>"}` and69 `with_monitored_hosts: true` to get the customer's group IDs. Skip for an70 all-customer sweep.712. `zabbix_problems_list` with `groupids`, `severities: [3,4,5]`,72 `sortfield: "eventid"`, `sortorder: "DESC"`, `selectTags: "extend"`,73 `selectSuppressionData: "extend"`, `limit: 200`. Add74 `symptom: false` to see only cause problems first.753. Collect the `objectid` values and call `zabbix_triggers_list` once with76 `triggerids`, `selectHosts: ["host","name"]`,77 `selectHostGroups: ["name"]`, `expandDescription: true` and78 `output: ["triggerid","description","priority","comments","url"]`. This maps79 every problem to its host and customer group in one call, and the trigger80 `comments` often carry the runbook note.814. If any problem came back on a host in maintenance or with82 `suppressed: 1`, confirm with `zabbix_maintenance_list` (`hostids`,83 `selectTimeperiods: "extend"`) before calling it noise.845. Rank: unacknowledged disaster/high first, then by age (`clock`), then85 problems shared by many hosts in one group (a site-wide outage beats ten86 single-host warnings).8788### Was anyone paged?8990`zabbix_alerts_list` with `eventids` set to the problem event IDs and91`selectHosts: ["name"]`. Each alert shows `sendto`, `status`92(0 not sent yet, 1 sent, 2 failed) and `error`. A disaster with no alert, or93with failed alerts, is an escalation gap to report, not just a problem.9495### Is it flapping?9697`zabbix_events_list` with `objectids: ["<triggerid>"]`,98`time_from` 24-72 hours back, `sortfield: "clock"`, `sortorder: "DESC"`,99`output: ["eventid","clock","value","severity"]`. Many 1/0 (problem/OK)100pairs in a short window means a flapping trigger: the fix is a threshold or101hysteresis change on the trigger, not a ticket per occurrence.102103### Currently-firing triggers (alternative view)104105`zabbix_triggers_list` with `only_true: true`, `monitored: true`,106`skipDependent: true`, `expandDescription: true`, `filter: {"value": 1}`,107`min_severity: 3`, `selectHosts: ["name"]`, `selectLastEvent: "extend"`.108`skipDependent` hides triggers whose parent (for example the site uplink)109is already in problem, which is usually what a NOC wants.110111## Gotchas112113- **Suppressed is not resolved.** Problems on hosts in maintenance are114 hidden in the frontend's default views but still returned unless you pass115 `suppressed: false`. Say which you included.116- **`recent: true` mixes in resolved problems.** Check `r_eventid` before117 calling something open.118- **Symptom problems inflate counts.** With cause and symptom linking,119 one router outage can own dozens of symptom problems. Count causes.120- **Severity is per problem, not per trigger.** Someone may have changed the121 problem's severity in the frontend; the trigger's `priority` keeps the122 original.123- **`acknowledged` is only Zabbix's view.** The ticket may live in the PSA124 without an acknowledgement in Zabbix. Do not report "nobody is working125 this" from `acknowledged: 0` alone.126- **Item values are not available.** `zabbix_items_list` and127 `zabbix_history_get` are opt-in and not listed in Conduit. The trigger128 name (with expanded macros) and `zabbix_trends_get` are what you have.129130## Output131132Group by customer (host group), then severity. Per problem: host, problem133name, severity, age, acknowledged yes/no, suppressed yes/no, alert sent /134failed / none, and a one-line next step. End with counts: open causes,135symptoms, suppressed, and any paging gaps.136137## Related Skills138139- [api-patterns](../api-patterns/SKILL.md) - Paging and parameters140- [monitoring-coverage](../monitoring-coverage/SKILL.md) - Silent proxies and gaps