Author Netdata Health Alerts
Use this skill before changing a health-alert definition. Alerts are production policy: a syntactically valid expression
can still page incorrectly, manufacture a recovery, duplicate an incident owner, or make a disappearing entity look healthy.
The normal goal when translating an alert from another system is Netdata-adapted operational equivalence, not
execution-engine emulation. Preserve the operator-visible incident as closely as Netdata can express it through NIDL
instances, alert beats, database lookups, native gap behavior, and chart obsoletion. Treat differences from the source
engine as explicit design facts to document and test, not as defects by default.
Read The Right Sources
- Read
AGENTS.md, the active SOW, the target alert file, and its collector/profile source.
- Read
docs/NIDL-Framework.md, src/health/REFERENCE.md, src/health/README.md, and
src/health/alert-configuration-ordering.md.
- Read the existing alert owner and search for duplicate names, contexts, metric prefixes, and equivalent generic alerts.
- When lifecycle, missing-data, expression, or lookup semantics matter, confirm them in current runtime source:
src/health/health_event_loop.c
src/health/health_variable.c
src/web/api/queries/query-execute.c
- the selected grouping implementation below
src/web/api/queries/
src/libnetdata/eval/eval-evaluate.c
- When the source is a go.d collector or Prometheus profile, also load the matching collector/profile skill and its required
references. Do not treat this skill as a replacement for collector or profile authoring guidance.
Never query or reconfigure a live Agent merely to validate an alert unless the user has explicitly authorized that access.
Establish The Alert Contract Before Editing
Write down the following in the active SOW before changing a non-trivial alert:
- Incident and owner: What real incident does this alert represent? Which one source owns it? Do not create a
product-named duplicate of an existing collection-failure, component-failure, or generic host alert just to change
routing or severity.
- Signal contract: Identify every source value and its meanings, including zero, non-zero, tri-state values, absent
dimensions, temporary collection failure, and known entity disappearance.
- Source intent and adaptation: Separate the source alert's operator intent from its engine syntax. Record which
condition, scope, severity, persistence intent, identity, and recovery semantics Netdata preserves; record every
deliberate difference. Use
NETDATA-ADAPTED as the normal classification. Reserve EXACT for a proven coincidental
match across timing, gaps, identity, recovery, and removal—not merely a similar expression.
- NIDL instance map: Record the monitored component, context, one instance type, RRDSET/chart ID identity, dimensions,
stable identity labels, current metadata labels, and the collector's obsoletion condition. An alert attaches to one
RRDSET/chart instance; a template applies that same rule independently to matching instances in one context.
- Identity: Choose an
alarm only for a specific chart instance; choose a template for a context-wide rule. An
identity label MUST identify the same monitored entity across its intended lifetime. A label whose source value can
change for that entity is current metadata, not RRDSET identity: preserve the chart ID and update/promote the label.
Confirm labels preserve the incident identity and do not create unbounded instances.
Prove whether each promoted source label is live metadata, creation-time metadata, or source-frozen history. Do not
describe a label as "current" merely because Netdata can update labels.
- Variables and aggregation: An unqualified variable resolves in the alert's local monitored component/instance before
broader candidates. Use a fully qualified chart/dimension reference only after proving the other chart is the matching
instance and labels select it unambiguously. Never synthesize an infrastructure-level alert by combining multiple
RRDSET instances in alert configuration; require a source-owned aggregate RRDSET at the higher component level instead.
The alert
component: classification field does not replace this NIDL mapping.
- Lifecycle: Specify expected
UNINITIALIZED, CLEAR, WARNING/CRITICAL, UNDEFINED, and REMOVED behavior.
State whether an ordinary recovery emits a zero or whether the chart/dimension disappears instead.
- Timing: State the source update cadence, explicit alert cadence, lookup window, source persistence intent, selected
Netdata adaptation, startup behavior, partial-gap behavior, stale-chart behavior, and notification policy. Do not call
a lookup window an implementation of another engine's
for: state machine.
- Validation: List the boundary, transition, missing-data, recovery, and duplicate-ownership tests that prove the
contract.
Pause for a user decision when the change creates a public alert contract, changes default notification policy, changes
the owner of an incident, needs shared health/query framework work, or cannot preserve the approved operator-visible
incident closely enough with Netdata-native behavior. A difference from Prometheus execution alone is not such a blocker.
Combine Values Across Dimensions, Charts, And Alerts
Choose the smallest pattern that can express the condition truthfully. Do not introduce cross-chart or intermediate-alert
plumbing when the required values already share the alert's chart.
Pattern 1 — dimensions on the alert's chart
Use unqualified dimension variables in calc, warn, or crit when every required value is a dimension on the chart
named by on::
template: example_ratio
on: app.state
calc: $total - $used
Runtime facts:
- Matching is by dimension ID or name in the alert's own chart.
- The default dimension value is
collector.last_stored_value: the latest database-stored value after interpolation, not
a database-window aggregate and not necessarily the raw last collector sample.
- A missing dimension makes the expression fail with an unknown variable and the alert value become
NaN.
- Prefer explicit non-finite guards when
UNDEFINED is the required missing-input state; do not rely on arithmetic or
comparisons to preserve NaN.
Use $dimension_raw only when the contract specifically needs the last collected value before interpolation/storage
presentation, and $dimension_last_collected_t only when it needs that dimension's collection timestamp.
Pattern 2 — dimensions from another chart or context
Use a dotted chart/context reference when values live on different charts:
template: example_cross_chart
on: app.usage
calc: $this * 100 / ${app.capacity.total}
The dotted prefix is interpreted from right to left as chart/context plus the final dimension name. Runtime resolution:
- An exact chart ID is checked first.
- A chart name may match.
- If the prefix is a context, every chart instance in that context contributes candidates for the final dimension.
- Every candidate is scored against the alert's chart labels by counting equal key/value labels.
- The candidate with the highest score wins. Equal scores are resolved by candidate traversal order, so avoid designs
where a tie is possible.
Consequences:
- This is the natural way to compare related chart instances, but the matching labels must make the intended instance
unique. A generic shared label such as only
component=ceph is usually insufficient across many cluster instances.
- Like Pattern 1, the selected dimension value is the latest stored value, not a query over a time window.
- Fully qualified names containing punctuation must use
${...} braces.
- A reference that resolves to no candidate fails as an unknown variable and produces
NaN.
Pattern 3 — an intermediate alert as a computed variable
Create a non-notifying helper alert when the required value itself needs a database lookup or multi-stage computation,
then reference that alert by name from the consuming alert:
template: example_window
on: app.work
lookup: average -1h of requests
calc: $this
to: silent
template: example_consumes
on: app.current
lookup: average -1m of requests
calc: $this / $example_window
warn: $this > 1
to: sysadmin
Runtime facts:
- Every running alert with the referenced name is a candidate.
- Candidates are selected by the same equal-label score used for cross-chart variables.
- The selected candidate contributes its current alert value (
rc->value), after that helper's own lookup and calc.
- This is the only one of the three patterns that can combine database-window results such as “max of the last hour of X
with the average of the last minute of Y”.
- Health evaluates all lookup/calculation phases before warning/critical phases, but helper snapshots are published as
each alert completes. A consumer with a different cadence can therefore read the helper's previous published value on
its first beat or after cadence drift. Align
every: or disclose this timing difference.
- Prefer
to: silent on helpers and document why they exist; a helper is instrumentation, not a second incident owner.
Choosing a pattern
- Same chart, latest values: Pattern 1.
- Different charts/contexts, latest values: Pattern 2.
- Any input needs a database window or a staged calculation: Pattern 3.
- Never use Pattern 3 merely to avoid a qualified dotted reference; its added timing and lifecycle coupling must earn its
place.
- Never combine multiple RRDSET instances into one infrastructure-level alert on the alert side. If no source-owned chart
provides the required aggregate, that is a collector/profile/framework gap, not an alert-expression workaround.
Model The Lifecycle Truthfully
| Source situation |
Alert-lifecycle consequence |
| Valid numeric input and both conditions are false |
CLEAR |
| Valid numeric input and warning/critical condition is true |
WARNING or CRITICAL |
| Some values exist in a lookup window and some are NULL |
The lookup continues using numeric values; NULL is not automatically a failed condition |
| Collector misses a collection while the chart remains live |
A runnable alert may evaluate a lookup based on stored data; no fresh sample is fabricated |
| Lookup runs and its selected window has no usable values |
$this is NaN; the final state is UNDEFINED only when the condition preserves it |
| A lookup's newest stored sample is too old for its runnable-history gate |
The health loop skips that evaluation; it does not manufacture CLEAR or guarantee an UNDEFINED transition |
| Collector knows the entity is gone and obsoletes the chart |
The alert enters REMOVED, not an ordinary zero/CLEAR recovery |
Important rules:
- A collection failure is not disappearance. Collectors MUST preserve a gap when they cannot measure and MUST obsolete
only an entity they know is gone. Do not add a false zero to make an alert recover.
- A chart can remain alert-eligible during a gap. The health loop skips an obsolete chart, but does not require a
newly collected value for every scheduled evaluation.
$last_collected_t and $update_every expose freshness when an
alert is the deliberate stale-collection owner.
- A live chart does not guarantee every lookup remains runnable forever. For a relative database lookup, the health
loop eventually skips evaluation when the newest stored point is too old for the requested window plus its bounded
update-interval tolerance. Skipping evaluation preserves the prior alert state; it is different from evaluating an
all-null result to
UNDEFINED.
- Obsoletion ends the instance alert. Once the collector knows an entity is gone and obsoletes its RRDSET, the health
rule no longer evaluates that instance. Do not attempt to emulate an infrastructure-level continuation in another
alert; collect a source-owned aggregate instance if that is the required product signal.
- A null result does not fabricate healthy state. The health loop assigns
NaN to an empty lookup result. A condition
that itself evaluates to NaN is UNDEFINED.
- Comparisons can consume
NaN. In the expression evaluator, NaN is false in boolean contexts and a comparison
such as $this == 0 produces a finite false result. That can yield CLEAR, not UNDEFINED. If the contract requires
UNDEFINED, make and test an expression path that preserves NaN; never assume a comparison does so.
- Some NULLs are not all NULLs. Query aggregation accepts numeric points and omits non-numeric ones. Do not assume a
partial collection gap invalidates a window unless the current runtime and the approved product contract say it does.
A general strict-coverage rule is shared-framework work, not an alert-file shortcut.
- Do not duplicate freshness ownership. Use
$now - $last_collected_t for an explicit collection/staleness alert only
when no existing generic collection-failure alert already owns the incident. A data-state alert normally owns the
measured condition, not the collector outage.
For a data-state alert that MUST become UNDEFINED when its lookup is non-finite, use and test this condition shape:
warn: ($this == nan or $this == inf) ? (nan) : (<numeric predicate>)
Use crit: in the same way. This preserves the non-finite result only; it does not turn a partial-null window into a
collection-failure alert or replace a known-obsolete chart's REMOVED lifecycle.
Adapt Timing To Netdata
Current State
Use calc when the source's current value is the full condition. Test start-up, missing input, normal recovery, and chart
obsoletion separately. delay: controls Netdata alert transition/notification hysteresis; it is not a general substitute
for a Prometheus for: duration.
Control Flapping With Three Combinable Layers
Stability is a signal-shaping problem, a threshold-boundary problem, and a transition-confirmation problem. Address them
deliberately in that order. Do not label every anti-flapping technique “hysteresis”: only delay: postpones an alert
transition.
Layer 1 — stabilize the queried value
Use lookup: to make $this a stable aggregate over an explicit observation window:
lookup: average -5m unaligned of latency
average smooths noisy utilization, rate, latency, and utilization-like signals.
min requires every observed numeric sample to remain active, appropriate for persisted binary fault states.
max preserves worst-case excursions when the incident is defined by peaks.
countif expresses percent-of-observed-time semantics.
This reduces noise but cannot prevent a stable aggregate from hovering near one threshold. A 5-minute average near 100 ms
can still cross a 100 ms threshold repeatedly.
Layer 2 — separate raise and clear thresholds
For policy thresholds whose signal may hover near the boundary, use a state-dependent predicate:
warn: $this > (($status >= $WARNING) ? (90) : (100))
crit: $this > (($status == $CRITICAL) ? (95) : (99))
This changes one threshold into two thresholds:
- clear-to-warning raises above 100;
- active-warning clears below 90;
- critical can independently use another raise/clear pair.
Important semantics:
- This is not hysteresis and does not delay any transition.
- The alert continues evaluating at its normal
every: cadence.
- It only changes the boundary used by the current status, so a genuine crossing of the recovery threshold acts
immediately.
- Preserve a non-finite guard around the complete expression when
UNDEFINED must remain possible.
Prefer this when independent raise/clear boundaries are meaningful and the signal is expected to linger near one boundary.
Do not use it to redefine a categorical exact condition into a policy band.
Layer 3 — confirm transitions with true hysteresis
Use delay: only when a transition must remain selected for a duration before Netdata executes the transition
notification:
delay: down 5m multiplier 1.5 max 1h
up delays a state escalation; down delays a recovery/de-escalation.
multiplier grows the delay when the state changes during the delay.
max caps the accumulated delay.
This is the only layer that postpones an alert transition notification. It can suppress rapid clear/reactivate
notification cycles, but it can also postpone a real transition. Use it sparingly and record the expected transition
delay in the alert contract.
Selection procedure
- Establish whether the incident is categorical, threshold policy, or derived arithmetic.
- Select the smallest truthful
lookup: window and aggregation first.
- For a noisy policy threshold, choose explicit raise/clear thresholds before adding transition delay.
- Add
delay: only when rapid transition notifications are independently harmful and later notification is acceptable.
- Test each layer: input noise, boundary crossing, recovery, reactivation, non-finite input, partial gap, and the exact
expected notification time.
Persistence Intent
Treat another system's for: D as an operator intent to suppress transient conditions. It is not a requirement to emulate
that system's pending-state machine. Choose the closest safe Netdata-native behavior from the source's actual numeric
state space, set an explicit every:, and disclose the differences:
- For a binary source where 1 means the active fault,
min -5m unaligned with an active predicate is true only when
every observed numeric value in the selected window is active. This is a Netdata observation-window adaptation, not
Prometheus for: 5m.
- For a binary source where 0 means the active fault, use the complementary aggregation/predicate that requires all
observed values to be zero; consult the lookup reference and prove it with state-sequence tests.
- For a tri-state or enumerated source, do not apply a binary
min/max rule by analogy. Use an exact predicate over the
full state space. For example, countif(!=target) returns the percentage of observed values outside target; zero means
every observed value matched.
min, max, and countif operate on the numeric samples they receive. Test active-to-other-state transitions in both
directions and record the intended partial-gap behavior.
- A new chart may become runnable with up to one chart update interval less history than the requested lookup window.
Therefore an already-active condition can raise up to roughly one source beat before
D after chart creation. Do not
conceal this with a hand-authored "window complete" flag.
- A missing sample does not reset an observation window. Values before and after a partial gap may contribute to the same
lookup. Collection-failure ownership remains separate unless the alert contract explicitly owns freshness.
- One health rule has one historical database lookup. A predicate combining multiple dimensions cannot gain exact
historical persistence by looking up one dimension and combining it with the others' current values. Prefer, in order:
a source-owned derived dimension when the persisted compound condition is essential; otherwise a truthful current-state
adaptation; never a mixed-time formula that changes the incident meaning.
- Do not lengthen the window mechanically by one assumed collector interval. Collector cadence is configurable, and that
does not repair partial gaps or create a source-engine pending state.
For every persistence adaptation, prove: chart startup, observed active history, each recovery state, each higher/lower
enumerated state, a partial source gap, a stale non-runnable lookup, collection resumption, chart obsoletion, and the
configured evaluation cadence. Never invent a pending state that Netdata does not expose, and never declare fidelity
from the look of an expression alone.
Keep Ownership And Identity Non-Duplicating
- Search existing stock alerts before adding one. Reuse an existing alert only when it really owns the same logical
incident; disclose routing/severity/lifecycle differences rather than silently relabeling it.
- Keep source collection failure, component/API collection failure, source data-state failure, and client-observed failure
as separate owners when they identify different operator actions.
- Filter templates with chart labels only when they select the intended RRDSET instance without changing its identity.
A current metadata label may be used as a filter only when the alert contract explicitly wants that current metadata;
do not turn it into
instances.by_labels merely to make filtering convenient. Exclude known named rules from generic
fallbacks, including special sources whose recovery is chart removal rather than zero.
- Preserve ordinary zero recovery. Do not convert an active-to-zero source into disappearance, and do not treat a
disappearing source as a normal CLEAR.
- Use the ordering guide for template/alarm precedence and user-versus-stock override behavior. Same-name definitions are
an override mechanism; different names coexist and can therefore duplicate incidents.
Validate The Actual Contract
Run the smallest relevant tests first, then the full affected suite. A complete alert change normally needs all applicable
items below:
- Run
/usr/sbin/netdata -W healthconfigtest for the built-in health parser and lookup suite.
- Add or update a focused test that reads the shipped alert template and asserts its context, labels, lookup, units,
cadence, expressions, routing, source ownership, and declared fidelity—not merely a copied expected string.
- Test the signal's lifecycle through the real query/health runtime where practical. If a lower-level deterministic model
is necessary, derive it directly from runtime timestamps and numeric-point selection; do not pass an arbitrary
windowComplete flag or label skipped NULLs as continuous evaluation. Cover startup, active transition, normal zero
recovery, a true all-null query, partial-null behavior, stale non-runnable behavior, collection resumption, known
disappearance/REMOVED, and label identity.
- For an expression that relies on
NaN, test the evaluator result directly. Check both direct NaN propagation and any
comparison/conditional branch; do not infer the result from ordinary floating-point intuition.
- Run collector/profile validation when the alert depends on a collector or profile change. Include source absence and
collision-bearing labels where relevant.
- Search for same-incident alerts, duplicate contexts, old names, fallback overlap, and generated artifacts. Record the
result in the SOW.
- For an externally sourced alert pack, commit a source-pinned mapping that records the original condition, scope,
severity, persistence intent, supported releases, Netdata owner, adaptation, and known differences. Tests MUST consume
that mapping rather than restating the intended result independently.
- Validate every published configuration example through the same job-construction prerequisites users need. In
particular, a job referencing
vnode: is incomplete unless the example defines or clearly links the required vnode.
- Run
git diff --check and the project-required validation/review gate before claiming completion.
healthconfigtest runs built-in health parser and lookup cases. It does not load every stock health template, prove that a
template attaches to the intended chart, or prove the runtime lifecycle. Cover those separately with source-aware contract
and transition tests.
Completion Check
Before requesting review, confirm all of the following:
- The alert has exactly one logical owner and a stable scope.
- Its NIDL instance map shows one monitored component and one instance-level alert target; any required aggregate is a
source-owned higher-level RRDSET, not an alert-side merge.
- Its source state values, gaps, absence, and recovery have been tested rather than assumed.
- Its
NaN, UNDEFINED, CLEAR, and REMOVED transitions match the recorded contract.
- Its timing is the closest safe Netdata adaptation, with startup/gap/stale differences disclosed;
delay: is not
standing in for another engine's persistence state machine.
- Notification defaults follow the approved product policy.
- No shared health/query behavior was added or assumed without the required separate scope approval.
Authoritative References
- Syntax, variables, lookups, and stock patterns:
src/health/REFERENCE.md
- State model and missing-data summary:
src/health/README.md
- NIDL component/instance/dimension/label model:
docs/NIDL-Framework.md
- Template/alarm and user/stock precedence:
src/health/alert-configuration-ordering.md
- Alert eligibility, lookup execution, and
REMOVED: src/health/health_event_loop.c
$now, $last_collected_t, $update_every, dimension freshness, same-chart variables, cross-chart/context
variables, alert variables, and label-score selection: src/health/health_variable.c
- Equal-label score implementation:
src/database/rrdlabels.c:rrdlabels_common_count()
- Query gaps and grouping:
src/web/api/queries/query-execute.c and the relevant grouping implementation
NaN expression semantics: src/libnetdata/eval/eval-evaluate.c
1---
2name: health-alert-authoring
3description: Author, adapt, modify, or review Netdata health alerts and alert templates. Use when translating alerts from another system; changing `src/health/health.d/*.conf`, lookup/calc/warn/crit expressions, lifecycle, timing, routing, ownership, or missing-data behavior; writing health-config tests; or selecting an alert's chart/context/label identity.
4metadata:
5 source: https://skillmd.com/skills/netdata/health-alert-authoring
6---
7
8# Author Netdata Health Alerts
9
10Use this skill before changing a health-alert definition. Alerts are production policy: a syntactically valid expression
11can still page incorrectly, manufacture a recovery, duplicate an incident owner, or make a disappearing entity look healthy.
12
13The normal goal when translating an alert from another system is **Netdata-adapted operational equivalence**, not
14execution-engine emulation. Preserve the operator-visible incident as closely as Netdata can express it through NIDL
15instances, alert beats, database lookups, native gap behavior, and chart obsoletion. Treat differences from the source
16engine as explicit design facts to document and test, not as defects by default.
17
18## Read The Right Sources
19
201. Read `AGENTS.md`, the active SOW, the target alert file, and its collector/profile source.
212. Read `docs/NIDL-Framework.md`, `src/health/REFERENCE.md`, `src/health/README.md`, and
22 `src/health/alert-configuration-ordering.md`.
233. Read the existing alert owner and search for duplicate names, contexts, metric prefixes, and equivalent generic alerts.
244. When lifecycle, missing-data, expression, or lookup semantics matter, confirm them in current runtime source:
25 - `src/health/health_event_loop.c`
26 - `src/health/health_variable.c`
27 - `src/web/api/queries/query-execute.c`
28 - the selected grouping implementation below `src/web/api/queries/`
29 - `src/libnetdata/eval/eval-evaluate.c`
305. When the source is a go.d collector or Prometheus profile, also load the matching collector/profile skill and its required
31 references. Do not treat this skill as a replacement for collector or profile authoring guidance.
32
33Never query or reconfigure a live Agent merely to validate an alert unless the user has explicitly authorized that access.
34
35## Establish The Alert Contract Before Editing
36
37Write down the following in the active SOW before changing a non-trivial alert:
38
39- **Incident and owner:** What real incident does this alert represent? Which one source owns it? Do not create a
40 product-named duplicate of an existing collection-failure, component-failure, or generic host alert just to change
41 routing or severity.
42- **Signal contract:** Identify every source value and its meanings, including zero, non-zero, tri-state values, absent
43 dimensions, temporary collection failure, and known entity disappearance.
44- **Source intent and adaptation:** Separate the source alert's operator intent from its engine syntax. Record which
45 condition, scope, severity, persistence intent, identity, and recovery semantics Netdata preserves; record every
46 deliberate difference. Use `NETDATA-ADAPTED` as the normal classification. Reserve `EXACT` for a proven coincidental
47 match across timing, gaps, identity, recovery, and removal—not merely a similar expression.
48- **NIDL instance map:** Record the monitored component, context, one instance type, RRDSET/chart ID identity, dimensions,
49 stable identity labels, current metadata labels, and the collector's obsoletion condition. An alert attaches to one
50 RRDSET/chart instance; a template applies that same rule independently to matching instances in one context.
51- **Identity:** Choose an `alarm` only for a specific chart instance; choose a `template` for a context-wide rule. An
52 identity label MUST identify the same monitored entity across its intended lifetime. A label whose source value can
53 change for that entity is current metadata, not RRDSET identity: preserve the chart ID and update/promote the label.
54 Confirm labels preserve the incident identity and do not create unbounded instances.
55 Prove whether each promoted source label is live metadata, creation-time metadata, or source-frozen history. Do not
56 describe a label as "current" merely because Netdata can update labels.
57- **Variables and aggregation:** An unqualified variable resolves in the alert's local monitored component/instance before
58 broader candidates. Use a fully qualified chart/dimension reference only after proving the other chart is the matching
59 instance and labels select it unambiguously. Never synthesize an infrastructure-level alert by combining multiple
60 RRDSET instances in alert configuration; require a source-owned aggregate RRDSET at the higher component level instead.
61 The alert `component:` classification field does not replace this NIDL mapping.
62- **Lifecycle:** Specify expected `UNINITIALIZED`, `CLEAR`, `WARNING`/`CRITICAL`, `UNDEFINED`, and `REMOVED` behavior.
63 State whether an ordinary recovery emits a zero or whether the chart/dimension disappears instead.
64- **Timing:** State the source update cadence, explicit alert cadence, lookup window, source persistence intent, selected
65 Netdata adaptation, startup behavior, partial-gap behavior, stale-chart behavior, and notification policy. Do not call
66 a lookup window an implementation of another engine's `for:` state machine.
67- **Validation:** List the boundary, transition, missing-data, recovery, and duplicate-ownership tests that prove the
68 contract.
69
70Pause for a user decision when the change creates a public alert contract, changes default notification policy, changes
71the owner of an incident, needs shared health/query framework work, or cannot preserve the approved operator-visible
72incident closely enough with Netdata-native behavior. A difference from Prometheus execution alone is not such a blocker.
73
74## Combine Values Across Dimensions, Charts, And Alerts
75
76Choose the smallest pattern that can express the condition truthfully. Do not introduce cross-chart or intermediate-alert
77plumbing when the required values already share the alert's chart.
78
79### Pattern 1 — dimensions on the alert's chart
80
81Use unqualified dimension variables in `calc`, `warn`, or `crit` when every required value is a dimension on the chart
82named by `on:`:
83
84```text
85template: example_ratio
86 on: app.state
87 calc: $total - $used
88```
89
90Runtime facts:
91
92- Matching is by dimension ID or name in the alert's own chart.
93- The default dimension value is `collector.last_stored_value`: the latest database-stored value after interpolation, not
94 a database-window aggregate and not necessarily the raw last collector sample.
95- A missing dimension makes the expression fail with an unknown variable and the alert value become `NaN`.
96- Prefer explicit non-finite guards when `UNDEFINED` is the required missing-input state; do not rely on arithmetic or
97 comparisons to preserve `NaN`.
98
99Use `$dimension_raw` only when the contract specifically needs the last collected value before interpolation/storage
100presentation, and `$dimension_last_collected_t` only when it needs that dimension's collection timestamp.
101
102### Pattern 2 — dimensions from another chart or context
103
104Use a dotted chart/context reference when values live on different charts:
105
106```text
107template: example_cross_chart
108 on: app.usage
109 calc: $this * 100 / ${app.capacity.total}
110```
111
112The dotted prefix is interpreted from right to left as chart/context plus the final dimension name. Runtime resolution:
113
1141. An exact chart ID is checked first.
1152. A chart name may match.
1163. If the prefix is a context, every chart instance in that context contributes candidates for the final dimension.
1174. Every candidate is scored against the alert's chart labels by counting equal key/value labels.
1185. The candidate with the highest score wins. Equal scores are resolved by candidate traversal order, so avoid designs
119 where a tie is possible.
120
121Consequences:
122
123- This is the natural way to compare related chart instances, but the matching labels must make the intended instance
124 unique. A generic shared label such as only `component=ceph` is usually insufficient across many cluster instances.
125- Like Pattern 1, the selected dimension value is the latest stored value, not a query over a time window.
126- Fully qualified names containing punctuation must use `${...}` braces.
127- A reference that resolves to no candidate fails as an unknown variable and produces `NaN`.
128
129### Pattern 3 — an intermediate alert as a computed variable
130
131Create a non-notifying helper alert when the required value itself needs a database lookup or multi-stage computation,
132then reference that alert by name from the consuming alert:
133
134```text
135template: example_window
136 on: app.work
137 lookup: average -1h of requests
138 calc: $this
139 to: silent
140
141template: example_consumes
142 on: app.current
143 lookup: average -1m of requests
144 calc: $this / $example_window
145 warn: $this > 1
146 to: sysadmin
147```
148
149Runtime facts:
150
151- Every running alert with the referenced name is a candidate.
152- Candidates are selected by the same equal-label score used for cross-chart variables.
153- The selected candidate contributes its current alert value (`rc->value`), after that helper's own lookup and `calc`.
154- This is the only one of the three patterns that can combine database-window results such as “max of the last hour of X
155 with the average of the last minute of Y”.
156- Health evaluates all lookup/calculation phases before warning/critical phases, but helper snapshots are published as
157 each alert completes. A consumer with a different cadence can therefore read the helper's previous published value on
158 its first beat or after cadence drift. Align `every:` or disclose this timing difference.
159- Prefer `to: silent` on helpers and document why they exist; a helper is instrumentation, not a second incident owner.
160
161### Choosing a pattern
162
163- Same chart, latest values: Pattern 1.
164- Different charts/contexts, latest values: Pattern 2.
165- Any input needs a database window or a staged calculation: Pattern 3.
166- Never use Pattern 3 merely to avoid a qualified dotted reference; its added timing and lifecycle coupling must earn its
167 place.
168- Never combine multiple RRDSET instances into one infrastructure-level alert on the alert side. If no source-owned chart
169 provides the required aggregate, that is a collector/profile/framework gap, not an alert-expression workaround.
170
171## Model The Lifecycle Truthfully
172
173| Source situation | Alert-lifecycle consequence |
174|---|---|
175| Valid numeric input and both conditions are false | `CLEAR` |
176| Valid numeric input and warning/critical condition is true | `WARNING` or `CRITICAL` |
177| Some values exist in a lookup window and some are NULL | The lookup continues using numeric values; NULL is not automatically a failed condition |
178| Collector misses a collection while the chart remains live | A runnable alert may evaluate a lookup based on stored data; no fresh sample is fabricated |
179| Lookup runs and its selected window has no usable values | `$this` is `NaN`; the final state is `UNDEFINED` only when the condition preserves it |
180| A lookup's newest stored sample is too old for its runnable-history gate | The health loop skips that evaluation; it does not manufacture `CLEAR` or guarantee an `UNDEFINED` transition |
181| Collector knows the entity is gone and obsoletes the chart | The alert enters `REMOVED`, not an ordinary zero/CLEAR recovery |
182
183Important rules:
184
185- **A collection failure is not disappearance.** Collectors MUST preserve a gap when they cannot measure and MUST obsolete
186 only an entity they know is gone. Do not add a false zero to make an alert recover.
187- **A chart can remain alert-eligible during a gap.** The health loop skips an obsolete chart, but does not require a
188 newly collected value for every scheduled evaluation. `$last_collected_t` and `$update_every` expose freshness when an
189 alert is the deliberate stale-collection owner.
190- **A live chart does not guarantee every lookup remains runnable forever.** For a relative database lookup, the health
191 loop eventually skips evaluation when the newest stored point is too old for the requested window plus its bounded
192 update-interval tolerance. Skipping evaluation preserves the prior alert state; it is different from evaluating an
193 all-null result to `UNDEFINED`.
194- **Obsoletion ends the instance alert.** Once the collector knows an entity is gone and obsoletes its RRDSET, the health
195 rule no longer evaluates that instance. Do not attempt to emulate an infrastructure-level continuation in another
196 alert; collect a source-owned aggregate instance if that is the required product signal.
197- **A null result does not fabricate healthy state.** The health loop assigns `NaN` to an empty lookup result. A condition
198 that itself evaluates to `NaN` is `UNDEFINED`.
199- **Comparisons can consume `NaN`.** In the expression evaluator, `NaN` is false in boolean contexts and a comparison
200 such as `$this == 0` produces a finite false result. That can yield `CLEAR`, not `UNDEFINED`. If the contract requires
201 `UNDEFINED`, make and test an expression path that preserves `NaN`; never assume a comparison does so.
202- **Some NULLs are not all NULLs.** Query aggregation accepts numeric points and omits non-numeric ones. Do not assume a
203 partial collection gap invalidates a window unless the current runtime and the approved product contract say it does.
204 A general strict-coverage rule is shared-framework work, not an alert-file shortcut.
205- **Do not duplicate freshness ownership.** Use `$now - $last_collected_t` for an explicit collection/staleness alert only
206 when no existing generic collection-failure alert already owns the incident. A data-state alert normally owns the
207 measured condition, not the collector outage.
208
209For a data-state alert that MUST become `UNDEFINED` when its lookup is non-finite, use and test this condition shape:
210
211```text
212warn: ($this == nan or $this == inf) ? (nan) : (<numeric predicate>)
213```
214
215Use `crit:` in the same way. This preserves the non-finite result only; it does not turn a partial-null window into a
216collection-failure alert or replace a known-obsolete chart's `REMOVED` lifecycle.
217
218## Adapt Timing To Netdata
219
220### Current State
221
222Use `calc` when the source's current value is the full condition. Test start-up, missing input, normal recovery, and chart
223obsoletion separately. `delay:` controls Netdata alert transition/notification hysteresis; it is not a general substitute
224for a Prometheus `for:` duration.
225
226## Control Flapping With Three Combinable Layers
227
228Stability is a signal-shaping problem, a threshold-boundary problem, and a transition-confirmation problem. Address them
229deliberately in that order. Do not label every anti-flapping technique “hysteresis”: only `delay:` postpones an alert
230transition.
231
232### Layer 1 — stabilize the queried value
233
234Use `lookup:` to make `$this` a stable aggregate over an explicit observation window:
235
236```text
237lookup: average -5m unaligned of latency
238```
239
240- `average` smooths noisy utilization, rate, latency, and utilization-like signals.
241- `min` requires every observed numeric sample to remain active, appropriate for persisted binary fault states.
242- `max` preserves worst-case excursions when the incident is defined by peaks.
243- `countif` expresses percent-of-observed-time semantics.
244
245This reduces noise but cannot prevent a stable aggregate from hovering near one threshold. A 5-minute average near 100 ms
246can still cross a 100 ms threshold repeatedly.
247
248### Layer 2 — separate raise and clear thresholds
249
250For policy thresholds whose signal may hover near the boundary, use a state-dependent predicate:
251
252```text
253warn: $this > (($status >= $WARNING) ? (90) : (100))
254crit: $this > (($status == $CRITICAL) ? (95) : (99))
255```
256
257This changes one threshold into two thresholds:
258
259- clear-to-warning raises above 100;
260- active-warning clears below 90;
261- critical can independently use another raise/clear pair.
262
263Important semantics:
264
265- This is **not hysteresis** and does not delay any transition.
266- The alert continues evaluating at its normal `every:` cadence.
267- It only changes the boundary used by the current status, so a genuine crossing of the recovery threshold acts
268 immediately.
269- Preserve a non-finite guard around the complete expression when `UNDEFINED` must remain possible.
270
271Prefer this when independent raise/clear boundaries are meaningful and the signal is expected to linger near one boundary.
272Do not use it to redefine a categorical exact condition into a policy band.
273
274### Layer 3 — confirm transitions with true hysteresis
275
276Use `delay:` only when a transition must remain selected for a duration before Netdata executes the transition
277notification:
278
279```text
280delay: down 5m multiplier 1.5 max 1h
281```
282
283- `up` delays a state escalation; `down` delays a recovery/de-escalation.
284- `multiplier` grows the delay when the state changes during the delay.
285- `max` caps the accumulated delay.
286
287This is the only layer that postpones an alert transition notification. It can suppress rapid clear/reactivate
288notification cycles, but it can also postpone a real transition. Use it sparingly and record the expected transition
289delay in the alert contract.
290
291### Selection procedure
292
2931. Establish whether the incident is categorical, threshold policy, or derived arithmetic.
2942. Select the smallest truthful `lookup:` window and aggregation first.
2953. For a noisy policy threshold, choose explicit raise/clear thresholds before adding transition delay.
2964. Add `delay:` only when rapid transition notifications are independently harmful and later notification is acceptable.
2975. Test each layer: input noise, boundary crossing, recovery, reactivation, non-finite input, partial gap, and the exact
298 expected notification time.
299
300### Persistence Intent
301
302Treat another system's `for: D` as an operator intent to suppress transient conditions. It is not a requirement to emulate
303that system's pending-state machine. Choose the closest safe Netdata-native behavior from the source's actual numeric
304state space, set an explicit `every:`, and disclose the differences:
305
306- For a binary source where **1 means the active fault**, `min -5m unaligned` with an active predicate is true only when
307 every **observed numeric value** in the selected window is active. This is a Netdata observation-window adaptation, not
308 Prometheus `for: 5m`.
309- For a binary source where **0 means the active fault**, use the complementary aggregation/predicate that requires all
310 observed values to be zero; consult the lookup reference and prove it with state-sequence tests.
311- For a tri-state or enumerated source, do not apply a binary `min`/`max` rule by analogy. Use an exact predicate over the
312 full state space. For example, `countif(!=target)` returns the percentage of observed values outside `target`; zero means
313 every observed value matched.
314- `min`, `max`, and `countif` operate on the numeric samples they receive. Test active-to-other-state transitions in both
315 directions and record the intended partial-gap behavior.
316- A new chart may become runnable with up to one chart update interval less history than the requested lookup window.
317 Therefore an already-active condition can raise up to roughly one source beat before `D` after chart creation. Do not
318 conceal this with a hand-authored "window complete" flag.
319- A missing sample does not reset an observation window. Values before and after a partial gap may contribute to the same
320 lookup. Collection-failure ownership remains separate unless the alert contract explicitly owns freshness.
321- One health rule has one historical database lookup. A predicate combining multiple dimensions cannot gain exact
322 historical persistence by looking up one dimension and combining it with the others' current values. Prefer, in order:
323 a source-owned derived dimension when the persisted compound condition is essential; otherwise a truthful current-state
324 adaptation; never a mixed-time formula that changes the incident meaning.
325- Do not lengthen the window mechanically by one assumed collector interval. Collector cadence is configurable, and that
326 does not repair partial gaps or create a source-engine pending state.
327
328For every persistence adaptation, prove: chart startup, observed active history, each recovery state, each higher/lower
329enumerated state, a partial source gap, a stale non-runnable lookup, collection resumption, chart obsoletion, and the
330configured evaluation cadence. Never invent a `pending` state that Netdata does not expose, and never declare fidelity
331from the look of an expression alone.
332
333## Keep Ownership And Identity Non-Duplicating
334
335- Search existing stock alerts before adding one. Reuse an existing alert only when it really owns the same logical
336 incident; disclose routing/severity/lifecycle differences rather than silently relabeling it.
337- Keep source collection failure, component/API collection failure, source data-state failure, and client-observed failure
338 as separate owners when they identify different operator actions.
339- Filter templates with chart labels only when they select the intended RRDSET instance without changing its identity.
340 A current metadata label may be used as a filter only when the alert contract explicitly wants that current metadata;
341 do not turn it into `instances.by_labels` merely to make filtering convenient. Exclude known named rules from generic
342 fallbacks, including special sources whose recovery is chart removal rather than zero.
343- Preserve ordinary zero recovery. Do not convert an active-to-zero source into disappearance, and do not treat a
344 disappearing source as a normal CLEAR.
345- Use the ordering guide for template/alarm precedence and user-versus-stock override behavior. Same-name definitions are
346 an override mechanism; different names coexist and can therefore duplicate incidents.
347
348## Validate The Actual Contract
349
350Run the smallest relevant tests first, then the full affected suite. A complete alert change normally needs all applicable
351items below:
352
3531. Run `/usr/sbin/netdata -W healthconfigtest` for the built-in health parser and lookup suite.
3542. Add or update a focused test that reads the shipped alert template and asserts its context, labels, lookup, units,
355 cadence, expressions, routing, source ownership, and declared fidelity—not merely a copied expected string.
3563. Test the signal's lifecycle through the real query/health runtime where practical. If a lower-level deterministic model
357 is necessary, derive it directly from runtime timestamps and numeric-point selection; do not pass an arbitrary
358 `windowComplete` flag or label skipped NULLs as continuous evaluation. Cover startup, active transition, normal zero
359 recovery, a true all-null query, partial-null behavior, stale non-runnable behavior, collection resumption, known
360 disappearance/`REMOVED`, and label identity.
3614. For an expression that relies on `NaN`, test the evaluator result directly. Check both direct `NaN` propagation and any
362 comparison/conditional branch; do not infer the result from ordinary floating-point intuition.
3635. Run collector/profile validation when the alert depends on a collector or profile change. Include source absence and
364 collision-bearing labels where relevant.
3656. Search for same-incident alerts, duplicate contexts, old names, fallback overlap, and generated artifacts. Record the
366 result in the SOW.
3677. For an externally sourced alert pack, commit a source-pinned mapping that records the original condition, scope,
368 severity, persistence intent, supported releases, Netdata owner, adaptation, and known differences. Tests MUST consume
369 that mapping rather than restating the intended result independently.
3708. Validate every published configuration example through the same job-construction prerequisites users need. In
371 particular, a job referencing `vnode:` is incomplete unless the example defines or clearly links the required vnode.
3729. Run `git diff --check` and the project-required validation/review gate before claiming completion.
373
374`healthconfigtest` runs built-in health parser and lookup cases. It does not load every stock health template, prove that a
375template attaches to the intended chart, or prove the runtime lifecycle. Cover those separately with source-aware contract
376and transition tests.
377
378## Completion Check
379
380Before requesting review, confirm all of the following:
381
382- The alert has exactly one logical owner and a stable scope.
383- Its NIDL instance map shows one monitored component and one instance-level alert target; any required aggregate is a
384 source-owned higher-level RRDSET, not an alert-side merge.
385- Its source state values, gaps, absence, and recovery have been tested rather than assumed.
386- Its `NaN`, `UNDEFINED`, `CLEAR`, and `REMOVED` transitions match the recorded contract.
387- Its timing is the closest safe Netdata adaptation, with startup/gap/stale differences disclosed; `delay:` is not
388 standing in for another engine's persistence state machine.
389- Notification defaults follow the approved product policy.
390- No shared health/query behavior was added or assumed without the required separate scope approval.
391
392## Authoritative References
393
394- Syntax, variables, lookups, and stock patterns: `src/health/REFERENCE.md`
395- State model and missing-data summary: `src/health/README.md`
396- NIDL component/instance/dimension/label model: `docs/NIDL-Framework.md`
397- Template/alarm and user/stock precedence: `src/health/alert-configuration-ordering.md`
398- Alert eligibility, lookup execution, and `REMOVED`: `src/health/health_event_loop.c`
399- `$now`, `$last_collected_t`, `$update_every`, dimension freshness, same-chart variables, cross-chart/context
400 variables, alert variables, and label-score selection: `src/health/health_variable.c`
401- Equal-label score implementation: `src/database/rrdlabels.c:rrdlabels_common_count()`
402- Query gaps and grouping: `src/web/api/queries/query-execute.c` and the relevant grouping implementation
403- `NaN` expression semantics: `src/libnetdata/eval/eval-evaluate.c`