Use when a harness, script or query has just produced a number you are about to act on — put in a PR body, use to rank options, or use to decide a change is unnecessary. A broken measurement almost never raises an error; it returns a plausible answer. So before trusting the instrument, run a probe whose correct output you already know and that the broken state CANNOT produce. Also check the three silent distortions: a suppressed error read as an absence, a window of N items read as a window of time, and a cross-product reported as coverage. Trigger terms: benchmark, A/B, measured, the numbers say, 0 hits, no difference, nothing found, last N commits, N x M, ranked by, harness, probe, sweep, no measurable cost, tests pass so it works.
A number is the most persuasive thing you can hand a reviewer, and a broken instrument is
the hardest failure to notice, because it does not fail. It returns something shaped
like an answer — often the answer you expected, which is worse. 0.000 vs 0.000 reads as
"no difference". 0 hits reads as "not present". +22% reads as a cost you had predicted.
None of them announces that the harness never loaded, the grep could not see the line, or
the two medians came from different sessions.
This is distinct from asking whether a number clears its noise floor
(state-the-noise-floor). That question assumes the
instrument works. This one asks the prior question: is this measuring what I think it
is measuring? A number can sit far above its noise floor and still be an artifact.
The remedy is cheap and specific: a control whose correct output you already know, chosen
so that the broken state and the working state give different answers.
When to use
A harness, script, benchmark or query has produced a figure you intend to publish, rank
by, or use to justify not making a change.
The result is reassuring — a tie, a zero, "nothing found", "no measurable difference".
Reassuring output deserves more suspicion, not less.
You are comparing two subjects and the comparison was assembled from separate runs.
You are about to report a count as coverage ("34 × 40 = 1360 comparisons").
When NOT to use
The observation is a single reproduced defect with a stack trace: it either crashes or
it does not, and the reproduction is the control.
The measurement's own tooling already ships a validated control you ran (a project's
benchmark harness with a self-check). Say which, rather than adding a second.
The number is not load-bearing — nothing changes whichever way it comes out. Then it
does not need a control; it needs deleting.
The practice (checklist)
Run a positive control FIRST, before reading any result. Not "does it run" — a
probe whose correct answer you already know. Done when you can state what the
control returned and why a broken harness could not have returned it.
Choose a control that can fail. The broken state and the working state must give
different answers. "Does the extension load?" is not a control if you never ask it;
"ask the baseline for a symbol only the branch has — it must be rejected" is.
Interleave, never batch, any A/B. Run A,B,A,B in one session. Medians from
separate sessions are not comparable, and that alone turned a reported +22% into
+2%.
Never suppress the instrument's errors.2>/dev/null converts "the instrument
never looked" into "the thing is not there" — the two situations that a
nothing-found result already cannot distinguish. Count failures separately and
report them; a zero with no error count beside it is not a measurement.
State the window in the unit you are comparing. "The last N items" carries a
hidden second variable — how long N items took — set by the subject itself. Convert
to calendar time before ranking anything, or rank on something else.
Do not report a cross-product as coverage.N × M is how many times the harness
ran, not how many times it tested. Measure the thing you assumed — did the knob
actually change the plan? — and report that instead.
Name the consumer before you start. Who reads this number, and what do they do
differently depending on its value? If you cannot answer, the measurement is not
cheap-and-harmless; it is the wrong work.
Re-run at the boundary. A verification claim is stateful: "the four tests fail
without the fix" can be true when written and a silent no-op by the time you post it,
because the fix got committed in between.
Rationalizations
Shortcut
Why it fails
"It ran without errors, so the numbers are good."
A broken measurement almost never errors. Four failures in one session on the same branch, in four different tools, every one returning a plausible number: a cross-session +22%, a 0.000-vs-0.000 tie from a harness that never loaded, a green suite from the wrong binary, and 0 corpus hits caused by line wrapping.
"A perfect tie means no difference."
It is also the exact output of an instrument measuring nothing. 0.000 vs 0.000 came from a harness where neither build had loaded — the entrypoint is derived from the filename, and both had been renamed. A tie is the single most reassuring result available, which is why it needs the harshest control.
"0 hits means the word isn't there."
Or the grep cannot see it. "one indexed field" returned 0 hits and sits at README:219, hard-wrapped across a line break. 0 was the expected shape of the answer, which is what made it invisible.
"The suite is green, so my change is loaded."
build/release/ is the same path whichever commit built it, and nothing announces which one did. Run a test that exists only on your branch; passing it is the proof, not the driver's mtime.
"2>/dev/null just keeps the output clean."
It also converted 35 of 77 failed network calls into 35 clean zeros. Every convenience that hides an error turns "never looked" into "not there".
"I compared the last 100 commits of each — same method both sides."
Same count, different duration: those windows spanned 1 day to 1,779 days across the repos being ranked. One number, two dimensions, and the second one is set by the thing you are measuring.
It is 1360 runs. Measuring the assumption — does each knob actually change the physical plan? — is what tells you how many were tests. The gap always flatters, and it is widest where the corpus is weakest.
RECEIPT
SOURCING-receipt — four measurement failures in one session, none of which raised
an error, on duckdb/duckdb-fts, 2026-08-12. Each returned a number that would have been
shipped:
#
what it returned
what was true
why it passed
1
standard queries +22% slower
+2% — the two medians came from different sessions
the delta was large and the direction was one I expected
2
A/B harness 0.000 vs 0.000
neither build ever loaded — the entrypoint derives from the filename, and both had been renamed
a perfect tie reads as "no difference"
3
probes run against the origin/main binary
the last make release had happened on a detached origin/main
build/release/ is the same path either way
4
"one indexed field" → 0 hits in their README
it is at README:219, hard-wrapped across a line break
0 is the expected shape of "not their word"
⇒ Failures 1 and 4 would have reached public text — a false cost figure in a PR body,
and a correct word cut from a README on false evidence. Failure 2 would have supported
"no measurable cost" in writing.
The one practice that catches all four is a control whose answer is known in advance:
ask each binary for a symbol only one of them has (a harness that loaded nothing fails
both; one that loaded the same build twice fails one); run a test that exists only on this
branch; grep a phrase you know is present.
Three further distortions, same class, measured separately:
A suppressed error becomes an absence — 2>/dev/null on a gh call turned 35 of
77 flaky-network failures into clean zeros in a count of review bodies carrying an
unanswered question.
A window of N items is not a window of time — ranking 50 repositories by distinct
outside authors "over the last 100 commits" compared windows spanning 1 day to 1,779
days.
A cross-product is not coverage — 34 knobs × 40 queries = 1360 went into a draft
PR comment as evidence; measuring whether each knob changed the physical plan is what
separates the runs from the tests.
Lifecycle
Signals it worked: the control fails when you deliberately break the harness; a
reviewer never asks "compared against what?"; and at least once the control catches an
artifact before it reaches prose — which is the only outcome that proves the practice
rather than the number.
What to log on a misfire: a control that passed in both the broken and working
states (it could not fail, so it was not a control), or a number that reached public
text and turned out to be an artifact — record which of the four shapes it was.
Record it in LEDGER.md.
Death criterion: none foreseeable for hand-built harnesses. Retire per-project if a
house ships a measurement harness with its own validated controls — then run theirs and
say so.
Relates to:state-the-noise-floor asks whether
a number clears its noise; this asks the prior question of whether the instrument
measured what you think. measure-before-you-widen
is this discipline applied to one specific decision (should the fix extend to a
neighbour). fuzz-before-you-claim-done attacks
the code with randomized inputs; this attacks your own instrument.
1---2name: run-a-control-before-you-trust-the-number3description: Use when a harness, script or query has just produced a number you are about to act on — put in a PR body, use to rank options, or use to decide a change is unnecessary. A broken measurement almost never raises an error; it returns a plausible answer. So before trusting the instrument, run a probe whose correct output you already know and that the broken state CANNOT produce. Also check the three silent distortions: a suppressed error read as an absence, a window of N items read as a window of time, and a cross-product reported as coverage. Trigger terms: benchmark, A/B, measured, the numbers say, 0 hits, no difference, nothing found, last N commits, N x M, ranked by, harness, probe, sweep, no measurable cost, tests pass so it works.4---56# Run a control before you trust the number78## Purpose910A number is the most persuasive thing you can hand a reviewer, and a broken instrument is11the hardest failure to notice, because **it does not fail**. It returns something shaped12like an answer — often the answer you expected, which is worse. `0.000 vs 0.000` reads as13"no difference". `0 hits` reads as "not present". `+22%` reads as a cost you had predicted.14None of them announces that the harness never loaded, the grep could not see the line, or15the two medians came from different sessions.1617This is distinct from asking whether a number clears its noise floor18([state-the-noise-floor](../state-the-noise-floor/SKILL.md)). That question assumes the19instrument works. This one asks the prior question: **is this measuring what I think it20is measuring?** A number can sit far above its noise floor and still be an artifact.2122The remedy is cheap and specific: a control whose correct output you already know, chosen23so that the broken state and the working state give *different* answers.2425## When to use2627- A harness, script, benchmark or query has produced a figure you intend to publish, rank28 by, or use to justify not making a change.29- The result is reassuring — a tie, a zero, "nothing found", "no measurable difference".30 Reassuring output deserves *more* suspicion, not less.31- You are comparing two subjects and the comparison was assembled from separate runs.32- You are about to report a count as coverage ("34 × 40 = 1360 comparisons").3334## When NOT to use3536- The observation is a single reproduced defect with a stack trace: it either crashes or37 it does not, and the reproduction *is* the control.38- The measurement's own tooling already ships a validated control you ran (a project's39 benchmark harness with a self-check). Say which, rather than adding a second.40- The number is not load-bearing — nothing changes whichever way it comes out. Then it41 does not need a control; it needs deleting.4243## The practice (checklist)4445- [ ] **Run a positive control FIRST, before reading any result.** Not "does it run" — a46 probe whose correct answer you already know. *Done when* you can state what the47 control returned and why a broken harness could not have returned it.48- [ ] **Choose a control that can fail.** The broken state and the working state must give49 *different* answers. "Does the extension load?" is not a control if you never ask it;50 "ask the baseline for a symbol only the branch has — it must be rejected" is.51- [ ] **Interleave, never batch, any A/B.** Run A,B,A,B in one session. Medians from52 separate sessions are not comparable, and that alone turned a reported **+22%** into53 **+2%**.54- [ ] **Never suppress the instrument's errors.** `2>/dev/null` converts "the instrument55 never looked" into "the thing is not there" — the two situations that a56 nothing-found result already cannot distinguish. Count failures separately and57 report them; a zero with no error count beside it is not a measurement.58- [ ] **State the window in the unit you are comparing.** "The last N items" carries a59 hidden second variable — how long N items took — set by the subject itself. Convert60 to calendar time before ranking anything, or rank on something else.61- [ ] **Do not report a cross-product as coverage.** `N × M` is how many times the harness62 *ran*, not how many times it *tested*. Measure the thing you assumed — did the knob63 actually change the plan? — and report that instead.64- [ ] **Name the consumer before you start.** Who reads this number, and what do they do65 differently depending on its value? If you cannot answer, the measurement is not66 cheap-and-harmless; it is the wrong work.67- [ ] **Re-run at the boundary.** A verification claim is stateful: "the four tests fail68 without the fix" can be true when written and a silent no-op by the time you post it,69 because the fix got committed in between.7071## Rationalizations7273| Shortcut | Why it fails |74|---|---|75| "It ran without errors, so the numbers are good." | A broken measurement almost never errors. Four failures in one session on the same branch, in four different tools, every one returning a plausible number: a cross-session +22%, a 0.000-vs-0.000 tie from a harness that never loaded, a green suite from the wrong binary, and 0 corpus hits caused by line wrapping. |76| "A perfect tie means no difference." | It is also the exact output of an instrument measuring nothing. `0.000 vs 0.000` came from a harness where neither build had loaded — the entrypoint is derived from the *filename*, and both had been renamed. A tie is the single most reassuring result available, which is why it needs the harshest control. |77| "0 hits means the word isn't there." | Or the grep cannot see it. `"one indexed field"` returned 0 hits and sits at README:219, hard-wrapped across a line break. 0 was the *expected* shape of the answer, which is what made it invisible. |78| "The suite is green, so my change is loaded." | `build/release/` is the same path whichever commit built it, and nothing announces which one did. Run a test that exists **only** on your branch; passing it is the proof, not the driver's mtime. |79| "`2>/dev/null` just keeps the output clean." | It also converted 35 of 77 failed network calls into 35 clean zeros. Every convenience that hides an error turns "never looked" into "not there". |80| "I compared the last 100 commits of each — same method both sides." | Same *count*, different *duration*: those windows spanned 1 day to 1,779 days across the repos being ranked. One number, two dimensions, and the second one is set by the thing you are measuring. |81| "34 knobs × 40 queries = 1360 comparisons is strong coverage." | It is 1360 *runs*. Measuring the assumption — does each knob actually change the physical plan? — is what tells you how many were tests. The gap always flatters, and it is widest where the corpus is weakest. |8283## RECEIPT8485***SOURCING-receipt*** — **four measurement failures in one session, none of which raised86an error**, on `duckdb/duckdb-fts`, 2026-08-12. Each returned a number that would have been87shipped:8889| # | what it returned | what was true | why it passed |90|---|---|---|---|91| 1 | standard queries **+22% slower** | **+2%** — the two medians came from different sessions | the delta was large and the direction was one I *expected* |92| 2 | A/B harness **0.000 vs 0.000** | neither build ever loaded — the entrypoint derives from the filename, and both had been renamed | a perfect tie reads as "no difference" |93| 3 | probes run against the **origin/main** binary | the last `make release` had happened on a detached `origin/main` | `build/release/` is the same path either way |94| 4 | `"one indexed field"` → **0 hits** in their README | it is at README:219, hard-wrapped across a line break | 0 is the expected shape of "not their word" |9596⇒ **Failures 1 and 4 would have reached public text** — a false cost figure in a PR body,97and a correct word cut from a README on false evidence. Failure 2 would have supported98"no measurable cost" in writing.99100**The one practice that catches all four** is a control whose answer is known in advance:101ask each binary for a symbol only one of them has (a harness that loaded nothing fails102both; one that loaded the same build twice fails one); run a test that exists only on this103branch; grep a phrase you *know* is present.104105**Three further distortions, same class, measured separately:**106107- **A suppressed error becomes an absence** — `2>/dev/null` on a `gh` call turned **35 of108 77** flaky-network failures into clean zeros in a count of review bodies carrying an109 unanswered question.110- **A window of N items is not a window of time** — ranking 50 repositories by distinct111 outside authors "over the last 100 commits" compared windows spanning **1 day to 1,779112 days**.113- **A cross-product is not coverage** — `34 knobs × 40 queries = 1360` went into a draft114 PR comment as evidence; measuring whether each knob changed the physical plan is what115 separates the runs from the tests.116117## Lifecycle118119- **Signals it worked:** the control fails when you deliberately break the harness; a120 reviewer never asks "compared against what?"; and at least once the control catches an121 artifact before it reaches prose — which is the only outcome that proves the practice122 rather than the number.123- **What to log on a misfire:** a control that passed in both the broken and working124 states (it could not fail, so it was not a control), or a number that reached public125 text and turned out to be an artifact — record which of the four shapes it was.126 Record it in [`LEDGER.md`](../../LEDGER.md).127- **Death criterion:** none foreseeable for hand-built harnesses. Retire per-project if a128 house ships a measurement harness with its own validated controls — then run theirs and129 say so.130- **Relates to:** [state-the-noise-floor](../state-the-noise-floor/SKILL.md) asks whether131 a number clears its noise; this asks the prior question of whether the instrument132 measured what you think. [measure-before-you-widen](../measure-before-you-widen/SKILL.md)133 is this discipline applied to one specific decision (should the fix extend to a134 neighbour). [fuzz-before-you-claim-done](../fuzz-before-you-claim-done/SKILL.md) attacks135 the code with randomized inputs; this attacks your own instrument.
Run npx skillmds@latest add serhiy-bzhezytskyy/run-a-control-before-you-trust-the-number in your terminal (requires Node.js), paste this page's agent-chat prompt into Claude, Cursor, or any MCP-connected agent, or download the SKILL.md file and copy it into your agent's skills directory.
Use when a harness, script or query has just produced a number you are about to act on — put in a PR body, use to rank options, or use to decide a change is unnecessary. A broken measurement almost never raises an error; it returns a plausible answer. So before trusting the instrument, run a probe whose correct output you already know and that the broken state CANNOT produce. Also check the three silent distortions: a suppressed error read as an absence, a window of N items read as a window of time, and a cross-product reported as coverage. Trigger terms: benchmark, A/B, measured, the numbers say, 0 hits, no difference, nothing found, last N commits, N x M, ranked by, harness, probe, sweep, no measurable cost, tests pass so it works. It is listed under Coding & Dev Tools on SkillMD.
This skill has not completed SkillMD's automated safety review yet. SkillMD never runs a skill's scripts for you; review the SKILL.md before installing.
This skill is tagged as working with Claude Code, Claude.ai, OpenAI Codex. SKILL.md is an open format, so most agents that read a skills directory can load it too.
Yes. Installing skills from SkillMD is free, and the skill stays under its author's original license.
serhiy-bzhezytskyy (@serhiy-bzhezytskyy) published this skill. Their other Agent Skills are listed on their SkillMD profile.
serhiy-bzhezytskyy