triage-agent-events
Private developer skill for triaging crashes, panics, and
fatals across the Netdata fleet. Reads the agent-events
systemd-journal namespace via the Netdata systemd-journal
Function (Cloud-proxied or direct-agent transport) and ships
scripts that bake in index-friendly query patterns.
Why this skill exists
40k-200k status events arrive on the ingestion server every
day on stable releases. The total fleet is 1.5M agents, so
the dataset is large and noisy (many unupdated agents report
crashes that have been fixed). Naive "grep all" queries are
slow and wasteful. This skill teaches the maintainer (and any
AI assistant helping them) how to slice the dataset
efficiently and how to interpret what comes back.
Workflow
+-------------------------+ +---------------------+
| get-events.sh | --> | <timestamp>.json |
| (cloud or agent API) | | in .local/audits |
+-------------------------+ +---------------------+
|
v
+------------------------+
| analyze-events.sh |
| --by signal|version| |
| function|... |
+------------------------+
|
v
+------------------------+
| cluster + read source |
| + fix the bug |
+------------------------+
The skill is a bug-investigation tool, not a generic logs
query tool. The two existing query-netdata-cloud and
query-netdata-agents skills already cover transport
mechanics; this skill EXTENDS them with the agent-events
specifics (what fields are present, what predicates are
index-friendly, what each enum value means for triage).
Key concepts (read first)
The dataset: 40k-200k status events / day on stable
releases, spread across 1.5M agents (not all restart
daily). Naive full-namespace queries with bare FTS are
slow.
Index-friendly queries (HARD RULE): use multi-value
field filters FIRST. The Netdata systemd-journal plugin
supports the syntax:
(FIELD1 in A, B, C) AND (FIELD2 in D, E, F) AND ...
Between fields = AND. Between values = OR. This is a
facet-engine feature, NOT raw journalctl. Use FTS via
query= only as a residual narrower over the structured
slice. See query-discipline.md.
Three transports (priority order):
- Cloud API -- proxied through Netdata Cloud at the
agent-events space. Primary for the team.
- Direct agent API -- against the agent-events node's
/api/v3/function?function=systemd-journal. Primary for
scripts.
- ssh to the host -- operator-only path; mentioned in
transports.md but no scripted ssh transport.
After-the-fact event model: agents POST events ONLY on
start (the previous session's exit reason). They commit
status to disk on start, stop, and at most every 10
minutes. So the meaningful query unit is "events posted in
the last 24 hours"; "the last hour" misses real crashes
that haven't restarted yet.
23h client-side dedup (src/daemon/status-file-dedup.c:11):
same agent + same event-content hash within 23h ->
suppressed at the producer. So 1 record per agent per
event-signature per day is the natural unit. Different
agents posting the same crash signature -> both arrive
(server does not dedup).
Default time + version filters: 24h time window;
latest stable + latest 2-3 nightlies for version. This
focuses triage on bugs that still matter. Wide windows
(--since '7d' or longer) are reserved for rare crashes
(1-per-few-days class) and for "when did this start /
get fixed" investigations.
AE_ field naming*: every JSON path in the producer's
status document becomes an AE_-prefixed journal field
(per log2journal --prefix 'AE_' on the ingestion server).
See AE_FIELDS.md for the verified map and enum meanings.
Table of contents
| Doc |
Purpose |
AE_FIELDS.md |
Verified field map (~80 rows) + enum meanings for triage. Indispensable. |
transports.md |
Cloud API + direct agent API call patterns; ssh footnote. |
update-cadence.md |
After-the-fact model, dedup, push timing, disk commits, query implications. |
query-discipline.md |
The multi-value filter syntax, structured-filters-first rule, anti-patterns. |
finding-crashes.md |
Recipe: signal crashes (SIGSEGV / SIGBUS / SIGFPE / SIGABRT) on stable. |
finding-fatals.md |
Recipe: deliberate fatals (OOM, disk full, asserts). |
recipes/INDEX.md |
Live catalog of recipes (find-by-function, find-by-version, find-related-to-work). |
how-tos/INDEX.md |
Live catalog: every analysis question becomes a how-to entry. |
Live how-to rule (mandatory)
If asked a concrete question about agent-events that requires
non-trivial analysis (multiple file reads, running queries,
cross-referencing with producer source) AND the answer is not
already documented in the per-domain guides above or in
recipes/, the assistant MUST author a new how-to under
how-tos/<slug>.md and add a one-line entry to
how-tos/INDEX.md BEFORE completing the task.
Scripts (in scripts/)
| Script |
Purpose |
_lib.sh |
Helpers (agentevents_* prefix). Sources query-netdata-agents/scripts/_lib.sh. Token-safe; ships a no-leak self-test. |
get-events.sh |
Fetch events of interest. Index-friendly defaults. JSON output to .local/audits/query-agent-events/. |
analyze-events.sh |
Group-by stats over a downloaded dump (signal, version, fatal_function, architecture, etc.). |
redact-events.sh |
Opt-in redaction (machine_guid / claim_id / host_id / ephemeral_id -> placeholders). For sharing only. |
Path discipline
This skill follows
<repo>/.agents/sensitive-data-discipline.md:
- Repo files: repo-relative (
<repo>/src/...).
- Sibling Netdata-org repos:
${NETDATA_REPOS_DIR}/<repo>/....
- agent-events host / namespace / machine GUID / node ID:
ALWAYS via env keys. Never literal values in any committed
file.
- Producer ingest URL: NEVER quoted literally. Reference only
as
src/daemon/status-file.c:988.
- Fetched event payloads land under
<repo>/.local/audits/query-agent-events/<timestamp>.json
(gitignored). Do NOT paste raw event JSON into committed
artifacts.
Required env keys
| Key |
Role |
NETDATA_CLOUD_TOKEN |
Cloud REST token (long-lived). |
NETDATA_CLOUD_HOSTNAME |
Cloud REST API host. |
AGENT_EVENTS_HOSTNAME |
Dual-duty: ssh host AND direct-HTTP host of the ingestion node. Can be IP or DNS name. NOT the journalctl namespace (hardcoded agent-events); NOT the Cloud room name (also hardcoded agent-events). |
AGENT_EVENTS_MACHINE_GUID |
Agent machine GUID for direct-agent transport. |
AGENT_EVENTS_NODE_ID |
Cloud node UUID for cloud-proxy transport. |
All values live in <repo>/.env (gitignored). See
<repo>/.agents/ENV.md for setup (where each value comes
from, sample formats, common mistakes).
Related skills
query-netdata-cloud -- transport: Cloud REST API.
query-netdata-agents -- transport: direct agent REST + bearer auto-mint.
- This skill consumes both via their
_lib.sh helpers.
1---
2name: triage-agent-events
3description: Bug-investigation tool for the Netdata agent-events ingestion namespace -- triage crashes, panics, fatals across the fleet by downloading events of interest and clustering locally. Covers the three transports (Cloud API and direct agent API are primary; ssh is operator-only), the verified AE_* field map and enum meanings, the dedup model (23h client-side per agent and event signature), the after-the-fact event timing (POST only on agent restart), and the Netdata systemd-journal plugin multi-value filter syntax (FIELD in A, B, C) AND ... Use when investigating crashes / panics / fatals; when grepping for events touching a specific function or file or version; when looking for regressions across versions; when an agent is reported crashing in a way you want to triage. Ships scripts get-events.sh and analyze-events.sh that fetch events with index-friendly filters and compute group-by stats. Defaults to last 24 hours and to the latest stable plus latest 2-3 nightlies.
4metadata:
5 source: https://skillmd.com/skills/netdata/triage-agent-events
6---
7
8# triage-agent-events
9
10Private developer skill for triaging crashes, panics, and
11fatals across the Netdata fleet. Reads the agent-events
12systemd-journal namespace via the Netdata `systemd-journal`
13Function (Cloud-proxied or direct-agent transport) and ships
14scripts that bake in index-friendly query patterns.
15
16## Why this skill exists
17
1840k-200k status events arrive on the ingestion server every
19day on stable releases. The total fleet is 1.5M agents, so
20the dataset is large and noisy (many unupdated agents report
21crashes that have been fixed). Naive "grep all" queries are
22slow and wasteful. This skill teaches the maintainer (and any
23AI assistant helping them) how to slice the dataset
24efficiently and how to interpret what comes back.
25
26## Workflow
27
28```
29+-------------------------+ +---------------------+
30| get-events.sh | --> | <timestamp>.json |
31| (cloud or agent API) | | in .local/audits |
32+-------------------------+ +---------------------+
33 |
34 v
35 +------------------------+
36 | analyze-events.sh |
37 | --by signal|version| |
38 | function|... |
39 +------------------------+
40 |
41 v
42 +------------------------+
43 | cluster + read source |
44 | + fix the bug |
45 +------------------------+
46```
47
48The skill is a bug-investigation tool, not a generic logs
49query tool. The two existing `query-netdata-cloud` and
50`query-netdata-agents` skills already cover transport
51mechanics; this skill EXTENDS them with the agent-events
52specifics (what fields are present, what predicates are
53index-friendly, what each enum value means for triage).
54
55## Key concepts (read first)
56
571. **The dataset**: 40k-200k status events / day on stable
58 releases, spread across 1.5M agents (not all restart
59 daily). Naive full-namespace queries with bare FTS are
60 slow.
61
622. **Index-friendly queries** (HARD RULE): use multi-value
63 field filters FIRST. The Netdata `systemd-journal` plugin
64 supports the syntax:
65 ```
66 (FIELD1 in A, B, C) AND (FIELD2 in D, E, F) AND ...
67 ```
68 Between fields = AND. Between values = OR. This is a
69 facet-engine feature, NOT raw journalctl. Use FTS via
70 `query=` only as a residual narrower over the structured
71 slice. See `query-discipline.md`.
72
733. **Three transports** (priority order):
74 - **Cloud API** -- proxied through Netdata Cloud at the
75 agent-events space. Primary for the team.
76 - **Direct agent API** -- against the agent-events node's
77 `/api/v3/function?function=systemd-journal`. Primary for
78 scripts.
79 - **ssh to the host** -- operator-only path; mentioned in
80 `transports.md` but no scripted ssh transport.
81
824. **After-the-fact event model**: agents POST events ONLY on
83 start (the previous session's exit reason). They commit
84 status to disk on start, stop, and at most every 10
85 minutes. So the meaningful query unit is "events posted in
86 the last 24 hours"; "the last hour" misses real crashes
87 that haven't restarted yet.
88
895. **23h client-side dedup** (`src/daemon/status-file-dedup.c:11`):
90 same agent + same event-content hash within 23h ->
91 suppressed at the producer. So 1 record per agent per
92 event-signature per day is the natural unit. Different
93 agents posting the same crash signature -> both arrive
94 (server does not dedup).
95
966. **Default time + version filters**: 24h time window;
97 latest stable + latest 2-3 nightlies for version. This
98 focuses triage on bugs that still matter. Wide windows
99 (`--since '7d'` or longer) are reserved for rare crashes
100 (1-per-few-days class) and for "when did this start /
101 get fixed" investigations.
102
1037. **AE_* field naming**: every JSON path in the producer's
104 status document becomes an `AE_`-prefixed journal field
105 (per `log2journal --prefix 'AE_'` on the ingestion server).
106 See `AE_FIELDS.md` for the verified map and enum meanings.
107
108## Table of contents
109
110| Doc | Purpose |
111|---|---|
112| `AE_FIELDS.md` | Verified field map (~80 rows) + enum meanings for triage. Indispensable. |
113| `transports.md` | Cloud API + direct agent API call patterns; ssh footnote. |
114| `update-cadence.md` | After-the-fact model, dedup, push timing, disk commits, query implications. |
115| `query-discipline.md` | The multi-value filter syntax, structured-filters-first rule, anti-patterns. |
116| `finding-crashes.md` | Recipe: signal crashes (SIGSEGV / SIGBUS / SIGFPE / SIGABRT) on stable. |
117| `finding-fatals.md` | Recipe: deliberate fatals (OOM, disk full, asserts). |
118| `recipes/INDEX.md` | Live catalog of recipes (find-by-function, find-by-version, find-related-to-work). |
119| `how-tos/INDEX.md` | Live catalog: every analysis question becomes a how-to entry. |
120
121## Live how-to rule (mandatory)
122
123If asked a concrete question about agent-events that requires
124non-trivial analysis (multiple file reads, running queries,
125cross-referencing with producer source) AND the answer is not
126already documented in the per-domain guides above or in
127`recipes/`, the assistant MUST author a new how-to under
128`how-tos/<slug>.md` and add a one-line entry to
129`how-tos/INDEX.md` BEFORE completing the task.
130
131## Scripts (in scripts/)
132
133| Script | Purpose |
134|---|---|
135| `_lib.sh` | Helpers (`agentevents_*` prefix). Sources `query-netdata-agents/scripts/_lib.sh`. Token-safe; ships a no-leak self-test. |
136| `get-events.sh` | Fetch events of interest. Index-friendly defaults. JSON output to `.local/audits/query-agent-events/`. |
137| `analyze-events.sh` | Group-by stats over a downloaded dump (signal, version, fatal_function, architecture, etc.). |
138| `redact-events.sh` | Opt-in redaction (machine_guid / claim_id / host_id / ephemeral_id -> placeholders). For sharing only. |
139
140## Path discipline
141
142This skill follows
143`<repo>/.agents/sensitive-data-discipline.md`:
144
145- Repo files: repo-relative (`<repo>/src/...`).
146- Sibling Netdata-org repos: `${NETDATA_REPOS_DIR}/<repo>/...`.
147- agent-events host / namespace / machine GUID / node ID:
148 ALWAYS via env keys. Never literal values in any committed
149 file.
150- Producer ingest URL: NEVER quoted literally. Reference only
151 as `src/daemon/status-file.c:988`.
152- Fetched event payloads land under
153 `<repo>/.local/audits/query-agent-events/<timestamp>.json`
154 (gitignored). Do NOT paste raw event JSON into committed
155 artifacts.
156
157## Required env keys
158
159| Key | Role |
160|---|---|
161| `NETDATA_CLOUD_TOKEN` | Cloud REST token (long-lived). |
162| `NETDATA_CLOUD_HOSTNAME` | Cloud REST API host. |
163| `AGENT_EVENTS_HOSTNAME` | Dual-duty: ssh host AND direct-HTTP host of the ingestion node. Can be IP or DNS name. NOT the journalctl namespace (hardcoded `agent-events`); NOT the Cloud room name (also hardcoded `agent-events`). |
164| `AGENT_EVENTS_MACHINE_GUID` | Agent machine GUID for direct-agent transport. |
165| `AGENT_EVENTS_NODE_ID` | Cloud node UUID for cloud-proxy transport. |
166
167All values live in `<repo>/.env` (gitignored). See
168`<repo>/.agents/ENV.md` for setup (where each value comes
169from, sample formats, common mistakes).
170
171## Related skills
172
173- `query-netdata-cloud` -- transport: Cloud REST API.
174- `query-netdata-agents` -- transport: direct agent REST + bearer auto-mint.
175- This skill consumes both via their `_lib.sh` helpers.