Query Contract Corpus — developer contract
tests/query-corpus/ is an end-to-end correctness suite for the query
engine of a stock netdata daemon. Fixtures enter through the real
streaming protocol and queries use the normal HTTP API. Structured responses
are strictly decoded before semantic checks; expectations are
fixture-derived, source-derived, or explicitly labeled stability/parity pins
under the classes below. tests/query-corpus/README.md describes L0-L11
plus cross-cutting API, options, and weights surfaces.
The suite is a self-contained Go module
(github.com/netdata/netdata/tests/query-corpus). Always run go commands
from inside tests/query-corpus/.
Correctness model — why this suite means something
The founding rule: semantic correctness expectations MUST be derived
independently of engine output. Capturing engine output is allowed only for
an explicitly labeled Class C stability pin; such a pin is not a correctness
oracle. Every check in the corpus belongs to one of three classes, with
different rules:
- Class A — first-principles oracles (the default). Fixtures are literal
Go definitions: charts, dimensions, and explicit
(timestamp, collected value, SN flags) points. Most deterministic
fixtures anchor at fixture.T0; wall-clock-only cases use bounded
envelopes. Expected values come from fixture arithmetic, conservation,
additivity, metadata laws, group-by folds, weights math, and anomaly-bit
counts. New checks MUST be Class A unless the transform genuinely cannot
be derived from first principles.
- Class B — ports of engine algorithms. Some transforms are engine
design decisions, not derivable math.
fixture/sn.go, fixture/tier.go,
fixture/timegroup.go, and fixture/viewpoints.go contain source-derived
ports; they also contain explicitly labeled Class A or contract helpers.
Class B code is written from the C source, never captured from engine
output. Rules:
- A port MUST cite
owner/repo @ commit plus the exact
repository-relative source path and line range it mirrors.
- Every divergence found between a port and the engine MUST be resolved
explicitly: either it is an engine bug (author a case for it) or an engine
quirk adopted into the oracle with a recorded pending ruling in the
manifest/SOW. Silently adjusting a port until the engine passes is the
cardinal sin of this suite ("fit-to-engine") and is prohibited.
- Where a port could hide drift, bound it independently (e.g. L2 pins the
SN quantization error envelope against the ORIGINAL values).
- A port MUST NOT be the oracle for a quantity that obeys an independent
law. Where one exists — conservation, monotonicity, additivity — that
law is derivable, so the check is Class A and the port has nothing to
contribute but the engine's own opinion.
ViewBuckets is a
source-derived model of the default tier-0 point-selection/interpolation
subset exercised by L9, not a port of the complete query executor. It is
appropriate only for those covered level-at-an-instant shapes.
ViewSumVolume is a Class A conservation oracle for volume and is
deliberately independent of that selection model. For every Class B
use, prefer an independent law when one exists.
- Fixtures MUST make the answer exact where the contract is exact. A
conservation check reads a difference, so anything else that moves the total
is noise the check will report as a defect. Above tier 0 the loudest such
source is resolution: when a window edge or a plan seam cuts a stored
record, the part inside can only be estimated from the record's width, and
with varying data that estimate is wrong by a share of the record — which
is what tiers cost, not a defect (see "Tiered rollups are not bugs").
- Grid-align the window so no edge falls inside a record, and where a
boundary is discovered rather than chosen (a rotated tier head), carry a
flat dimension alongside the varying ones: a constant survives rollup
exactly, so its share of a cut record is its exact truth and any
difference left is arithmetic.
- Worked example: CASE-026's seam contract first read 0.49s of data
"missing" across a plan switch, entirely from the seam cutting one
random-valued tier1 record. On the flat dimension the same three windows
answer 7,200,000 to the digit at every zoom. Had the noisy version been
committed, the corpus would have reported accepted rollup as an engine
bug — the mirror image of fit-to-engine, and just as damaging.
- Class C — stability pins and parity checks. Formatter byte pins
(primarily L7 and selected option tests) detect output-contract changes;
parity checks prove internal coherence only. Neither establishes
first-principles correctness. Rules:
- A Class C pin MUST be paired with independent validity checks where
they exist (e.g. "the payload parses as JSON", "values equal the
fixture-derived numbers inside the pinned envelope").
- Updating a pinned byte string requires a justified contract change
(a PR that deliberately changes the output format), never "the test
started failing".
Falsifiability discipline: when an expectation and the engine disagree,
there are exactly two exits — the engine is wrong (the case stays as written
and joins the broken list until it is fixed) or a recorded ruling says the
behavior is intended (the case is rewritten to assert the ruled behavior,
with the quirk documented). There is no third option where the oracle is
quietly bent to match, and none where the disagreement is filed away as
acceptable.
Architecture map
stream/stream.go — the fixture child. Speaks plugins.d over the
streaming socket: CHART/DIMENSION/CLABEL, live samples
(BEGIN2/SET2/END2), v1 paced samples (BEGIN/SET/END), and replication
(RBEGIN/RSET/REND). Protocol words quote-switch per word (qw()):
plugins.d accepts both ' and " delimiters, so ids carrying an
apostrophe ship double-quoted. SET2 sends the value explicitly because
the # shorthand truncates fractional values on the parser side. Exact
parser citations are listed under "Changing oracles, pins, and the
harness."
daemon/daemon.go — the harness. Resolves the binary and source tree as a
pair, boots the stock binary with a scratch run dir, assigns a unique
daemon identity, waits for that identity through the HTTP API, exposes
query helpers and WaitRetention, and uses bounded shutdown.
fixture/ — the fixture model plus source-derived Class B ports and
explicitly labeled Class A or contract helpers.
canon/ — the strict typed json2 decoder and point-column helpers. It
rejects invalid schema indices, widths, labels, field types/ranges, and
non-finite numeric metadata while decoding nullable value/hidden cells.
Query-specific assertion helpers enforce whether those nulls are semantically
valid and carry the required annotations.
*_test.go — layer*.go ladder tests, cross-cutting surface tests, and
per-bug caseNNN_test.go files.
manifest.json + MANIFEST.md — the ledger data and its human-readable
mirror (below). manifest.go embeds and validates the data. Keep all three
in sync in the same commit.
reference-python/ — local-only cross-check implementation. It is NOT
tracked and MUST NOT be committed.
The manifest
Every contract case has an entry in manifest.json; its proves, cloud, and
optional fixed_by fields are mirrored as a row in MANIFEST.md.
- One contract key MUST represent one independently actionable semantic
invariant. Multiple fixtures or inputs MAY share a key when they all prove
that same invariant. Independent claims, such as numeric value correctness
and unit rendering, MUST use separate keys and separately registered test or
subtest scopes. A shared green/red verdict is prohibited because an existing
failure in one claim can hide a new regression in another while the broken
contract roster remains unchanged.
- Register ordinary Go assertions with
trackContract(t, name) before any
operation that may fail or skip. Its cleanup records Error, Fatal, and
Skip.
- Use
assertContract(t, name, held) when the test computes an explicit
contract verdict. Call registerContract(t, name) before shared work, so
an earlier Fatal or Skip leaves the verdict visibly incomplete instead
of letting a default-true accumulator report it green.
- When independent test scopes jointly prove one contract, declare their
names in
ManifestCase.Components and register each with
trackContractComponent. One component passing never substitutes for
another component that did not run.
A broken contract fails. Always. On master, on a feature branch,
whether or not the break is already known.
- The manifest records NO expected outcome. There is no "known broken, and
therefore fine" state, and adding one is prohibited: it makes a broken
query engine report success, and this suite exists to name what is
broken, not to keep a list of exceptions.
- An unfiltered root-package run ends with the deduplicated list of broken
contracts and fails if any manifest contract or required component did
not run. That complete list is the corpus's answer — the open-defect
list produced by measurement rather than by hand.
- A filtered daemon-backed run reports how many contracts were fully evaluated
and never claims the complete corpus holds. The deliberately named
daemon-free harness/unit fast path prints only its Go test result and no
query-contract verdict.
-list also prints no contract verdict.
go test ./... therefore exits non-zero while any contract is broken or
the ledger is incomplete. That is the intended signal, not a problem to
suppress. The corpus is not wired into CI.
A case name is <layer-or-CASE-id>/<slug>; Proves is one sentence a
maintainer can read as the contract claim. A case whose bug is fixed keeps
its test as the regression guard and records FixedBy: "#PR".
Running
- Build the daemon first:
ninja -C build netdata from the repo root. By
default the suite pairs ../../build/netdata with ../../src.
- For another checkout, set both paths:
QUERY_CORPUS_NETDATA=/absolute/path/to/build/netdata QUERY_CORPUS_SRC=/absolute/path/to/src go test .... One-sided overrides
are rejected. The pair is operator-declared provenance; the harness does
not inspect binary build metadata.
- Full suite:
cd tests/query-corpus && go test ./... -count=1. Expect
several minutes; duration is hardware and case-selection dependent.
- One test:
go test -count=1 -run 'TestName' .. Some tests consume a
shared palette authored by an earlier layer; include that fixture-producing
test in the filter or run the full suite when a test reports that its
palette is unavailable. A prerequisite skip is not correctness evidence.
- Keep the daemon run dir for inspection:
QUERY_CORPUS_KEEP=1 (it is
always kept on failure; the path is printed as daemon run dir kept:).
- Capture the verdict honestly:
go test ... ; echo "exit=$?" — piping
through tail masks the exit code.
- Before every push of the corpus branch, run the full suite and compare
the broken list to the previous run. It must not grow, and no case may
break that was holding before. The list being non-empty is expected while
the query engine still has open defects.
Authoring fixtures
- Epoch: all points anchor at
fixture.T0. For update_every > 1,
pre-align the series: base := fixture.T0 - fixture.T0%int64(ue) —
storage keeps pushed timestamps exactly, but views re-grid onto absolute
update_every multiples, so unaligned fixtures make expectations
needlessly hard.
- Host GUIDs:
guid(n) builds a deterministic machine GUID. n MUST
be unique across the whole suite — hosts persist in the shared daemon for
the entire run, so a collision silently cross-contaminates two tests.
Before taking a number, grep -n 'guid(' *_test.go and pick an unused
range; ranges used by loops (e.g. soak attempts) reserve their whole
span.
- Settle discipline: after pushing, block on
td.WaitRetention(...)
before querying. Ordinary helpers keep the connection open through
assertions to isolate storage/query checks from teardown timing. CASE-015
deliberately closes immediately and guards the #23118 delivered-data
drain guarantee; immediate close is no longer documented as data loss.
- Weights fixtures: rrdcontexts stamps retention ~1–2s after chart
creation; weights queries return empty until then. Settle on the
contexts
first_time_t (see weightsSettle), not only on retention.
- Tolerances: exact comparison is the default.
Chart.ValueTolerance
is ONLY for quantization-probing fixtures, with the reason in a comment.
- Tier window alignment:
TierWindows(gran) keys on ABSOLUTE multiples
of the granularity, not on offsets from T0. fixture.T0 % 60 == 20, so
the first tier-1 window ends at T0+40 and a fixture whose shape is
keyed on the sample index straddles two regimes per stored window. Anchor
tier queries at T0+40, and let the oracle — never the fixture's index
arithmetic — say what each window contains.
- Forcing wide-point re-delivery: ask for a view grid FINER than the
stored data (
DataParamsTier(ctx, 1, after, before, buckets, ...) with
buckets a multiple of the stored window count). Each stored point is
then delivered to several buckets, carrying its original start and an
INTERPOLATED value — so any grouping that reads value instead of the
window's own statistics answers differently per bucket. That is the only
way to reach the repeat path from a query, and it is how
CASE-023/tier-wide-point caught a constant window being judged on an
interpolated blend of two windows.
Adding a case
- Author the fixture (Class A first; reach for a Class B oracle only when
the transform requires it).
- Push it (
pushLiveBurst for live bursts; paced v1 or replication where
the ingestion path is the thing under test), settle, query.
- Compute expectations in Go from the fixture definition. Never paste a
number you got from the engine.
- Add one manifest entry and
MANIFEST.md row per independently actionable
semantic invariant, then register each contract at the narrowest
test/subtest scope that proves it. Do not put independent value, units,
metadata, or formatting claims behind one green/red verdict.
- Run the full suite; a new case MUST NOT destabilize existing cases
(watch for GUID collisions and shared-host mutations).
Adding a case for a bug (bug workflow)
A case for a known bug is written exactly like any other case: it states
the CORRECT behavior and fails while the engine gets it wrong. It is not
marked, excused, or inverted — it joins the broken list until the fix
lands, and the broken list is what the corpus is for.
- Reproduce the divergence deterministically in its own
caseNNN_test.go
with a minimal fixture. The check asserts the CORRECT behavior and feeds
the result into assertContract.
- Add the manifest entry with a
Proves sentence stating the contract
precisely (what correct is), not the bug's symptoms.
- Confirm it fails on today's daemon, and that the failure names the real
defect — a case that fails for the wrong reason is worse than none.
- The fix goes in its OWN branch/PR — never mixed into the corpus branch.
- Validate the fix branch against the corpus before opening the PR:
- build the fix branch, save the binary aside;
- from the corpus checkout:
QUERY_CORPUS_NETDATA=<fix-checkout>/build/netdata QUERY_CORPUS_SRC=<fix-checkout>/src go test -count=1 -run '<the case plus neighboring pins>' .
Both paths MUST describe the same operator-declared checkout.
- the case MUST now hold, and every other case that was holding MUST
still hold (zero collateral).
- When the fix merges: rebase the corpus branch onto the merge, record
FixedBy: "#PR", reword the case comment and Proves to describe the
contract in force, run the full suite, push. The case lives on as the
regression guard.
- If the divergence is ruled intended behavior instead: change the case to
assert the ruled behavior, document the quirk in the oracle comment and
the
Proves text, and record the ruling.
Changing oracles, pins, and the harness
- An oracle change MUST cite its justification: the fixture math (Class A)
or the C source being ported (Class B). "It makes the suite pass" is not
a justification — that is fit-to-engine.
- A Class B port correction that changes expected values MUST state which
divergence prompted it and why it is not an engine bug.
- Byte-pins change only with a deliberate output-contract change.
- Determinism: expectations MUST NOT depend on wall-clock time. Tests that
must touch "now" (live edge, relative windows) assert ENVELOPES (bounded
ranges, row-count bounds), not exact values.
- Protocol emitters (
stream/) mirror the parser's actual grammar. At
netdata/netdata @ 043f50ec075441010c1495250871d37a8ac69f8d, the
authoritative surfaces are:
- quote/token splitting:
src/libnetdata/line_splitter/line_splitter.h:34-119;
BEGIN2: src/plugins.d/pluginsd_parser.c:815-960;
SET2: src/plugins.d/pluginsd_parser.c:963-1125;
END2: src/plugins.d/pluginsd_parser.c:1128-1163;
- A/R/E flags:
src/plugins.d/pluginsd_internals.h:461-485;
RBEGIN/RSET/REND:
src/plugins.d/pluginsd_replication.c:112-215,217-280,367-441.
Extend emitters only from the relevant parser path and update the checked
revision/ranges when the port changes.
Known boundaries (extension points, not history)
Deliberately out of scope so far; extending into them is welcome and each
states what it takes:
- KS2 exact tail values: the ks2 weights oracle pins the engine's
special cases; a full KSfbar port would make every ks2 weight exact.
- Natural-points full oracle: natural mode pins count/values and a
two-candidate boundary check; a full oracle needs the natural-mode point
walk ported.
- 64-bit counter wrap: unreachable through the signed text protocol;
needs a different ingestion vector.
- Float collected values on the reset path: the v1 SET path parses
integers; the reset/overflow pins use integer counters only.
points > 86400: the API caps points; oversized-grid behavior is
unpinned.
- Cloud tier: replaying the raw halves of L5/L6 through the real cloud
aggregator is designed but lives outside this repo.
Gotchas
- Run
go commands from tests/query-corpus/ (own module); running from
the repo root fails with "go.mod not found".
- Values print through the engine's number formatter: a stored
22.000000000000004 prints as 22. Compare parsed numbers, not
strings, unless the check IS a byte-pin.
- The default shared daemon serves most tests. Tests that need isolation,
rotation, or restarts boot dedicated daemons; never restart or reconfigure
the default shared daemon from an unrelated test.
- After system library upgrades, rebuild
build/netdata before blaming a
test failure on the suite.
- IDE diagnostics on the Go files can be stale;
go vet ./... is the
authority.
1---
2name: tests-query-corpus
3description: Developer contract for the query contract corpus (tests/query-corpus) — the black-box correctness suite for the Netdata query engine. Use when running the corpus, adding or extending corpus cases, authoring fixtures, changing an oracle or a byte-pin, adding a case for a query-engine bug, recording the fixing PR after it merges, or validating a query-engine fix branch against the corpus.
4metadata:
5 source: https://skillmd.com/skills/netdata/tests-query-corpus
6---
7
8# Query Contract Corpus — developer contract
9
10`tests/query-corpus/` is an end-to-end correctness suite for the query
11engine of a stock `netdata` daemon. Fixtures enter through the real
12streaming protocol and queries use the normal HTTP API. Structured responses
13are strictly decoded before semantic checks; expectations are
14fixture-derived, source-derived, or explicitly labeled stability/parity pins
15under the classes below. `tests/query-corpus/README.md` describes L0-L11
16plus cross-cutting API, options, and weights surfaces.
17
18The suite is a self-contained Go module
19(`github.com/netdata/netdata/tests/query-corpus`). Always run `go` commands
20from inside `tests/query-corpus/`.
21
22## Correctness model — why this suite means something
23
24The founding rule: **semantic correctness expectations MUST be derived
25independently of engine output.** Capturing engine output is allowed only for
26an explicitly labeled Class C stability pin; such a pin is not a correctness
27oracle. Every check in the corpus belongs to one of three classes, with
28different rules:
29
30- **Class A — first-principles oracles (the default).** Fixtures are literal
31 Go definitions: charts, dimensions, and explicit
32 `(timestamp, collected value, SN flags)` points. Most deterministic
33 fixtures anchor at `fixture.T0`; wall-clock-only cases use bounded
34 envelopes. Expected values come from fixture arithmetic, conservation,
35 additivity, metadata laws, group-by folds, weights math, and anomaly-bit
36 counts. New checks MUST be Class A unless the transform genuinely cannot
37 be derived from first principles.
38- **Class B — ports of engine algorithms.** Some transforms are engine
39 design decisions, not derivable math. `fixture/sn.go`, `fixture/tier.go`,
40 `fixture/timegroup.go`, and `fixture/viewpoints.go` contain source-derived
41 ports; they also contain explicitly labeled Class A or contract helpers.
42 Class B code is written from the C source, never captured from engine
43 output. Rules:
44 - A port MUST cite `owner/repo @ commit` plus the exact
45 repository-relative source path and line range it mirrors.
46 - Every divergence found between a port and the engine MUST be resolved
47 explicitly: either it is an engine bug (author a case for it) or an engine
48 quirk adopted into the oracle **with a recorded pending ruling** in the
49 manifest/SOW. Silently adjusting a port until the engine passes is the
50 cardinal sin of this suite ("fit-to-engine") and is prohibited.
51 - Where a port could hide drift, bound it independently (e.g. L2 pins the
52 SN quantization error envelope against the ORIGINAL values).
53 - A port MUST NOT be the oracle for a quantity that obeys an independent
54 law. Where one exists — conservation, monotonicity, additivity — that
55 law is derivable, so the check is Class A and the port has nothing to
56 contribute but the engine's own opinion. `ViewBuckets` is a
57 source-derived model of the default tier-0 point-selection/interpolation
58 subset exercised by L9, not a port of the complete query executor. It is
59 appropriate only for those covered level-at-an-instant shapes.
60 `ViewSumVolume` is a Class A conservation oracle for volume and is
61 deliberately independent of that selection model. For every Class B
62 use, prefer an independent law when one exists.
63- **Fixtures MUST make the answer exact where the contract is exact.** A
64 conservation check reads a difference, so anything else that moves the total
65 is noise the check will report as a defect. Above tier 0 the loudest such
66 source is resolution: when a window edge or a plan seam cuts a stored
67 record, the part inside can only be estimated from the record's width, and
68 with varying data that estimate is wrong by a share of the record — which
69 is what tiers cost, not a defect (see "Tiered rollups are not bugs").
70 - Grid-align the window so no edge falls inside a record, and where a
71 boundary is discovered rather than chosen (a rotated tier head), carry a
72 **flat dimension** alongside the varying ones: a constant survives rollup
73 exactly, so its share of a cut record is its exact truth and any
74 difference left is arithmetic.
75 - Worked example: CASE-026's seam contract first read 0.49s of data
76 "missing" across a plan switch, entirely from the seam cutting one
77 random-valued tier1 record. On the flat dimension the same three windows
78 answer 7,200,000 to the digit at every zoom. Had the noisy version been
79 committed, the corpus would have reported accepted rollup as an engine
80 bug — the mirror image of fit-to-engine, and just as damaging.
81- **Class C — stability pins and parity checks.** Formatter byte pins
82 (primarily L7 and selected option tests) detect output-contract changes;
83 parity checks prove internal coherence only. Neither establishes
84 first-principles correctness. Rules:
85 - A Class C pin MUST be paired with independent validity checks where
86 they exist (e.g. "the payload parses as JSON", "values equal the
87 fixture-derived numbers inside the pinned envelope").
88 - Updating a pinned byte string requires a justified contract change
89 (a PR that deliberately changes the output format), never "the test
90 started failing".
91
92**Falsifiability discipline:** when an expectation and the engine disagree,
93there are exactly two exits — the engine is wrong (the case stays as written
94and joins the broken list until it is fixed) or a recorded ruling says the
95behavior is intended (the case is rewritten to assert the ruled behavior,
96with the quirk documented). There is no third option where the oracle is
97quietly bent to match, and none where the disagreement is filed away as
98acceptable.
99
100## Architecture map
101
102- `stream/stream.go` — the fixture child. Speaks plugins.d over the
103 streaming socket: `CHART`/`DIMENSION`/`CLABEL`, live samples
104 (`BEGIN2/SET2/END2`), v1 paced samples (`BEGIN/SET/END`), and replication
105 (`RBEGIN/RSET/REND`). Protocol words quote-switch per word (`qw()`):
106 plugins.d accepts both `'` and `"` delimiters, so ids carrying an
107 apostrophe ship double-quoted. `SET2` sends the value explicitly because
108 the `#` shorthand truncates fractional values on the parser side. Exact
109 parser citations are listed under "Changing oracles, pins, and the
110 harness."
111- `daemon/daemon.go` — the harness. Resolves the binary and source tree as a
112 pair, boots the stock binary with a scratch run dir, assigns a unique
113 daemon identity, waits for that identity through the HTTP API, exposes
114 query helpers and `WaitRetention`, and uses bounded shutdown.
115- `fixture/` — the fixture model plus source-derived Class B ports and
116 explicitly labeled Class A or contract helpers.
117- `canon/` — the strict typed json2 decoder and point-column helpers. It
118 rejects invalid schema indices, widths, labels, field types/ranges, and
119 non-finite numeric metadata while decoding nullable value/hidden cells.
120 Query-specific assertion helpers enforce whether those nulls are semantically
121 valid and carry the required annotations.
122- `*_test.go` — `layer*.go` ladder tests, cross-cutting surface tests, and
123 per-bug `caseNNN_test.go` files.
124- `manifest.json` + `MANIFEST.md` — the ledger data and its human-readable
125 mirror (below). `manifest.go` embeds and validates the data. Keep all three
126 in sync in the same commit.
127- `reference-python/` — local-only cross-check implementation. It is NOT
128 tracked and MUST NOT be committed.
129
130## The manifest
131
132Every contract case has an entry in `manifest.json`; its `proves`, `cloud`, and
133optional `fixed_by` fields are mirrored as a row in `MANIFEST.md`.
134
135- **One contract key MUST represent one independently actionable semantic
136 invariant.** Multiple fixtures or inputs MAY share a key when they all prove
137 that same invariant. Independent claims, such as numeric value correctness
138 and unit rendering, MUST use separate keys and separately registered test or
139 subtest scopes. A shared green/red verdict is prohibited because an existing
140 failure in one claim can hide a new regression in another while the broken
141 contract roster remains unchanged.
142- Register ordinary Go assertions with `trackContract(t, name)` before any
143 operation that may fail or skip. Its cleanup records `Error`, `Fatal`, and
144 `Skip`.
145- Use `assertContract(t, name, held)` when the test computes an explicit
146 contract verdict. Call `registerContract(t, name)` before shared work, so
147 an earlier `Fatal` or `Skip` leaves the verdict visibly incomplete instead
148 of letting a default-true accumulator report it green.
149- When independent test scopes jointly prove one contract, declare their
150 names in `ManifestCase.Components` and register each with
151 `trackContractComponent`. One component passing never substitutes for
152 another component that did not run.
153
154**A broken contract fails. Always.** On master, on a feature branch,
155whether or not the break is already known.
156
157- The manifest records NO expected outcome. There is no "known broken, and
158 therefore fine" state, and adding one is prohibited: it makes a broken
159 query engine report success, and this suite exists to name what is
160 broken, not to keep a list of exceptions.
161- An unfiltered root-package run ends with the deduplicated list of broken
162 contracts and fails if any manifest contract or required component did
163 not run. **That complete list is the corpus's answer** — the open-defect
164 list produced by measurement rather than by hand.
165- A filtered daemon-backed run reports how many contracts were fully evaluated
166 and never claims the complete corpus holds. The deliberately named
167 daemon-free harness/unit fast path prints only its Go test result and no
168 query-contract verdict. `-list` also prints no contract verdict.
169- `go test ./...` therefore exits non-zero while any contract is broken or
170 the ledger is incomplete. That is the intended signal, not a problem to
171 suppress. The corpus is not wired into CI.
172
173A case name is `<layer-or-CASE-id>/<slug>`; `Proves` is one sentence a
174maintainer can read as the contract claim. A case whose bug is fixed keeps
175its test as the regression guard and records `FixedBy: "#PR"`.
176
177## Running
178
179- Build the daemon first: `ninja -C build netdata` from the repo root. By
180 default the suite pairs `../../build/netdata` with `../../src`.
181- For another checkout, set both paths:
182 `QUERY_CORPUS_NETDATA=/absolute/path/to/build/netdata
183 QUERY_CORPUS_SRC=/absolute/path/to/src go test ...`. One-sided overrides
184 are rejected. The pair is operator-declared provenance; the harness does
185 not inspect binary build metadata.
186- Full suite: `cd tests/query-corpus && go test ./... -count=1`. Expect
187 several minutes; duration is hardware and case-selection dependent.
188- One test: `go test -count=1 -run 'TestName' .`. Some tests consume a
189 shared palette authored by an earlier layer; include that fixture-producing
190 test in the filter or run the full suite when a test reports that its
191 palette is unavailable. A prerequisite skip is not correctness evidence.
192- Keep the daemon run dir for inspection: `QUERY_CORPUS_KEEP=1` (it is
193 always kept on failure; the path is printed as `daemon run dir kept:`).
194- Capture the verdict honestly: `go test ... ; echo "exit=$?"` — piping
195 through `tail` masks the exit code.
196- **Before every push of the corpus branch, run the full suite and compare
197 the broken list to the previous run.** It must not grow, and no case may
198 break that was holding before. The list being non-empty is expected while
199 the query engine still has open defects.
200
201## Authoring fixtures
202
203- **Epoch**: all points anchor at `fixture.T0`. For `update_every > 1`,
204 pre-align the series: `base := fixture.T0 - fixture.T0%int64(ue)` —
205 storage keeps pushed timestamps exactly, but views re-grid onto absolute
206 `update_every` multiples, so unaligned fixtures make expectations
207 needlessly hard.
208- **Host GUIDs**: `guid(n)` builds a deterministic machine GUID. `n` MUST
209 be unique across the whole suite — hosts persist in the shared daemon for
210 the entire run, so a collision silently cross-contaminates two tests.
211 Before taking a number, `grep -n 'guid(' *_test.go` and pick an unused
212 range; ranges used by loops (e.g. soak attempts) reserve their whole
213 span.
214- **Settle discipline**: after pushing, block on `td.WaitRetention(...)`
215 before querying. Ordinary helpers keep the connection open through
216 assertions to isolate storage/query checks from teardown timing. CASE-015
217 deliberately closes immediately and guards the #23118 delivered-data
218 drain guarantee; immediate close is no longer documented as data loss.
219- **Weights fixtures**: rrdcontexts stamps retention ~1–2s after chart
220 creation; weights queries return empty until then. Settle on the
221 contexts `first_time_t` (see `weightsSettle`), not only on retention.
222- **Tolerances**: exact comparison is the default. `Chart.ValueTolerance`
223 is ONLY for quantization-probing fixtures, with the reason in a comment.
224- **Tier window alignment**: `TierWindows(gran)` keys on ABSOLUTE multiples
225 of the granularity, not on offsets from `T0`. `fixture.T0 % 60 == 20`, so
226 the first tier-1 window ends at `T0+40` and a fixture whose shape is
227 keyed on the sample index straddles two regimes per stored window. Anchor
228 tier queries at `T0+40`, and let the oracle — never the fixture's index
229 arithmetic — say what each window contains.
230- **Forcing wide-point re-delivery**: ask for a view grid FINER than the
231 stored data (`DataParamsTier(ctx, 1, after, before, buckets, ...)` with
232 `buckets` a multiple of the stored window count). Each stored point is
233 then delivered to several buckets, carrying its original start and an
234 INTERPOLATED value — so any grouping that reads `value` instead of the
235 window's own statistics answers differently per bucket. That is the only
236 way to reach the repeat path from a query, and it is how
237 CASE-023/tier-wide-point caught a constant window being judged on an
238 interpolated blend of two windows.
239
240## Adding a case
241
2421. Author the fixture (Class A first; reach for a Class B oracle only when
243 the transform requires it).
2442. Push it (`pushLiveBurst` for live bursts; paced v1 or replication where
245 the ingestion path is the thing under test), settle, query.
2463. Compute expectations in Go from the fixture definition. Never paste a
247 number you got from the engine.
2484. Add one manifest entry and `MANIFEST.md` row per independently actionable
249 semantic invariant, then register each contract at the narrowest
250 test/subtest scope that proves it. Do not put independent value, units,
251 metadata, or formatting claims behind one green/red verdict.
2525. Run the full suite; a new case MUST NOT destabilize existing cases
253 (watch for GUID collisions and shared-host mutations).
254
255## Adding a case for a bug (bug workflow)
256
257A case for a known bug is written exactly like any other case: it states
258the CORRECT behavior and fails while the engine gets it wrong. It is not
259marked, excused, or inverted — it joins the broken list until the fix
260lands, and the broken list is what the corpus is for.
261
2621. Reproduce the divergence deterministically in its own `caseNNN_test.go`
263 with a minimal fixture. The check asserts the CORRECT behavior and feeds
264 the result into `assertContract`.
2652. Add the manifest entry with a `Proves` sentence stating the contract
266 precisely (what correct is), not the bug's symptoms.
2673. Confirm it fails on today's daemon, and that the failure names the real
268 defect — a case that fails for the wrong reason is worse than none.
2694. The fix goes in its OWN branch/PR — never mixed into the corpus branch.
2705. Validate the fix branch against the corpus before opening the PR:
271 - build the fix branch, save the binary aside;
272 - from the corpus checkout:
273 `QUERY_CORPUS_NETDATA=<fix-checkout>/build/netdata
274 QUERY_CORPUS_SRC=<fix-checkout>/src go test -count=1 -run '<the case
275 plus neighboring pins>' .`
276 Both paths MUST describe the same operator-declared checkout.
277 - the case MUST now hold, and every other case that was holding MUST
278 still hold (zero collateral).
2796. When the fix merges: rebase the corpus branch onto the merge, record
280 `FixedBy: "#PR"`, reword the case comment and `Proves` to describe the
281 contract in force, run the full suite, push. The case lives on as the
282 regression guard.
2837. If the divergence is ruled intended behavior instead: change the case to
284 assert the ruled behavior, document the quirk in the oracle comment and
285 the `Proves` text, and record the ruling.
286
287## Changing oracles, pins, and the harness
288
289- An oracle change MUST cite its justification: the fixture math (Class A)
290 or the C source being ported (Class B). "It makes the suite pass" is not
291 a justification — that is fit-to-engine.
292- A Class B port correction that changes expected values MUST state which
293 divergence prompted it and why it is not an engine bug.
294- Byte-pins change only with a deliberate output-contract change.
295- Determinism: expectations MUST NOT depend on wall-clock time. Tests that
296 must touch "now" (live edge, relative windows) assert ENVELOPES (bounded
297 ranges, row-count bounds), not exact values.
298- Protocol emitters (`stream/`) mirror the parser's actual grammar. At
299 `netdata/netdata @ 043f50ec075441010c1495250871d37a8ac69f8d`, the
300 authoritative surfaces are:
301 - quote/token splitting:
302 `src/libnetdata/line_splitter/line_splitter.h:34-119`;
303 - `BEGIN2`: `src/plugins.d/pluginsd_parser.c:815-960`;
304 - `SET2`: `src/plugins.d/pluginsd_parser.c:963-1125`;
305 - `END2`: `src/plugins.d/pluginsd_parser.c:1128-1163`;
306 - A/R/E flags: `src/plugins.d/pluginsd_internals.h:461-485`;
307 - `RBEGIN`/`RSET`/`REND`:
308 `src/plugins.d/pluginsd_replication.c:112-215,217-280,367-441`.
309 Extend emitters only from the relevant parser path and update the checked
310 revision/ranges when the port changes.
311
312## Known boundaries (extension points, not history)
313
314Deliberately out of scope so far; extending into them is welcome and each
315states what it takes:
316
317- **KS2 exact tail values**: the ks2 weights oracle pins the engine's
318 special cases; a full KSfbar port would make every ks2 weight exact.
319- **Natural-points full oracle**: natural mode pins count/values and a
320 two-candidate boundary check; a full oracle needs the natural-mode point
321 walk ported.
322- **64-bit counter wrap**: unreachable through the signed text protocol;
323 needs a different ingestion vector.
324- **Float collected values on the reset path**: the v1 SET path parses
325 integers; the reset/overflow pins use integer counters only.
326- **`points` > 86400**: the API caps points; oversized-grid behavior is
327 unpinned.
328- **Cloud tier**: replaying the raw halves of L5/L6 through the real cloud
329 aggregator is designed but lives outside this repo.
330
331## Gotchas
332
333- Run `go` commands from `tests/query-corpus/` (own module); running from
334 the repo root fails with "go.mod not found".
335- Values print through the engine's number formatter: a stored
336 `22.000000000000004` prints as `22`. Compare parsed numbers, not
337 strings, unless the check IS a byte-pin.
338- The default shared daemon serves most tests. Tests that need isolation,
339 rotation, or restarts boot dedicated daemons; never restart or reconfigure
340 the default shared daemon from an unrelated test.
341- After system library upgrades, rebuild `build/netdata` before blaming a
342 test failure on the suite.
343- IDE diagnostics on the Go files can be stale; `go vet ./...` is the
344 authority.