Review changes under evals/specs/** and evals/worlds/** for one question:
does this diff let a spec pass while the specific behavior it claims to test
is broken?
Test code has a different purpose from production code. Review the validity
of its evidence, not production hardening, abstraction, style, or preferred
helper usage. Channel conventions in evals/README.md are authoring guidance;
a channel mismatch alone is not a finding.
Report a MEDIUM (advisory) finding only when ALL of these hold:
- The changed lines introduce or materially worsen the gap.
- The spec bypasses the behavior under test or asserts evidence unrelated to
that behavior.
- You can identify a concrete broken behavior that would still pass, grounded
in the spec and relevant implementation. A hypothetical possibility or the
availability of a different helper is insufficient.
Examples worth reporting:
- A spec claims a person can submit a form, but directly invokes the API and
never submits through the UI, so broken form wiring is not exercised.
- A spec claims a save persists data, but only asserts the seeded value and
never observes the result of saving.
- A spec claims visible success, but checks an internal success flag while
the implementation demonstrably never renders the result.
Do not report:
- Read-only DOM/CDP inspection,
evaluateOnSurface, document.body.innerText,
or probe.* merely because user.see/user.notSee could be used instead.
For example, opening /pricing and asserting new prices in rendered body
text is acceptable pricing evidence; the title saying "visitors see" does
not by itself require a different helper. Report only if the implementation
shows that the asserted text does not prove the specific claimed outcome.
seed.*, direct API calls, or browser evaluation used to arrange state,
including setup between actions; report only when setup substitutes for
the behavior actually under test.agent.* in specs testing the agent, control rail, or voice.- Missing
// TODO(primitive): comments or helper migration suggestions.
- Test-only shortcuts, mocks, or fixtures that do not invalidate the claim.
- Pre-existing gaps, title wording alone, or anything outside the scoped paths.
Never report high or low. Each finding must quote the claimed behavior,
identify the changed line that bypasses it, explain the concrete failure that
would still pass, and suggest the smallest fix. If that evidence is missing,
report nothing.
1---2name: spec-provenance-review3description: Flag concrete false-positive proof introduced by changed specs, not test helper or channel preferences. Advisory only; never gates Warden clearance.4---5
6Review changes under `evals/specs/**` and `evals/worlds/**` for one question:
7does this diff let a spec pass while the specific behavior it claims to test
8is broken?
9
10Test code has a different purpose from production code. Review the validity
11of its evidence, not production hardening, abstraction, style, or preferred
12helper usage. Channel conventions in `evals/README.md` are authoring guidance;
13a channel mismatch alone is not a finding.
14
15Report a MEDIUM (advisory) finding only when ALL of these hold:
16
17- The changed lines introduce or materially worsen the gap.
18- The spec bypasses the behavior under test or asserts evidence unrelated to
19 that behavior.
20- You can identify a concrete broken behavior that would still pass, grounded
21 in the spec and relevant implementation. A hypothetical possibility or the
22 availability of a different helper is insufficient.
23
24Examples worth reporting:
25
26- A spec claims a person can submit a form, but directly invokes the API and
27 never submits through the UI, so broken form wiring is not exercised.
28- A spec claims a save persists data, but only asserts the seeded value and
29 never observes the result of saving.
30- A spec claims visible success, but checks an internal success flag while
31 the implementation demonstrably never renders the result.
32
33Do not report:
34
35- Read-only DOM/CDP inspection, `evaluateOnSurface`, `document.body.innerText`,
36 or `probe.*` merely because `user.see`/`user.notSee` could be used instead.
37 For example, opening `/pricing` and asserting new prices in rendered body
38 text is acceptable pricing evidence; the title saying "visitors see" does
39 not by itself require a different helper. Report only if the implementation
40 shows that the asserted text does not prove the specific claimed outcome.
41- `seed.*`, direct API calls, or browser evaluation used to arrange state,
42 including setup between actions; report only when setup substitutes for
43 the behavior actually under test.
44- `agent.*` in specs testing the agent, control rail, or voice.
45- Missing `// TODO(primitive):` comments or helper migration suggestions.
46- Test-only shortcuts, mocks, or fixtures that do not invalidate the claim.
47- Pre-existing gaps, title wording alone, or anything outside the scoped paths.
48
49Never report `high` or `low`. Each finding must quote the claimed behavior,
50identify the changed line that bypasses it, explain the concrete failure that
51would still pass, and suggest the smallest fix. If that evidence is missing,
52report nothing.
gabrielmoreira