falsegreen-skill: false-positive test detection
This skill is a thin entry point. Default path: judge from this file. It carries
the protocol summary, the structural code index, the S-series table, and the
look-alike exemptions, which covers a Mode A review of a TypeScript, JavaScript,
or Robot Framework test end to end.
Open a plugin-root file only where this file sends you:
../../reference.md - the full definition or worked example of one specific
code, after the index below has named it. Read the passage, not the section.
../../SKILL.md - the long-form J1-J6 wording, the Python structural catalog,
Mode B (authoring), or Mode C (AI-fix). At ~36 KiB on a ~32 KiB host this
replaces the rest of your context, so it is a decision, not a warm-up read.
Do not open either before reading this file through.
Mandatory for every language, Python included: load the semantic catalog. The
S-series (S1-S18, S21) is language-agnostic and lives in reference.md under
## Patterns only the semantic pass can catch (AI-only), above the per-language
sections. The root SKILL.md names only S3 and S17, so a run that loads a
language section alone never sees the rest. The complete table is in this file
under "Semantic cases", with the look-alike exemptions right after it. Use both:
the S-series without its exemptions produces false positives, which this skill
treats as worse than a miss. ../../reference.md has the same codes as prose with
examples if a finding needs the long form.
Mandatory for any non-Python file (JavaScript, TypeScript, Robot Framework,
Gherkin, Tavern): work the structural code index below before you judge. It
names every code all three scanners emit, so it is what stops a non-Python review
under-detecting. The root SKILL.md tables are complete only for Python; for every
other language they are a summary.
Reach for the matching reference.md section only when the index has named a
candidate and you need its full definition, and then read that passage, not the
section. On a ~32 KiB host a whole section does not fit beside this file: the TS/JS
section alone is ~19 KiB.
This file is the floor: the structural code index names every code all three
scanners emit, the semantic table carries a row per S-code, and the exemptions
follow it. ../../reference.md is ~92 KiB, so an eager full read overruns the
budget and truncates mid-file without warning, which is why the order above is
index first and passage second. ../../fragments/precision-rules.md carries the
precision rules if the host resolves plugin-root paths.
Structural code index (all three scanners)
Generated from schema/code-catalog.json and schema/scanner-codes.json, so it
cannot drift. Scanner column: py falsegreen, js falsegreen-js, rf
falsegreen-robot. Severity - means no fixed severity. Diagnostic codes
(D-series, M2) are opt-in: apply them only when asked.
One limit worth knowing before you match on a row. A code shared across languages
carries one title here, and for the C-series that title is the Python form. C31 is
the clearest case: the row reads "capsys.readouterr() result discarded", and the
Robot form of the same id is any captured keyword value that is never asserted on.
C3, C9, C23 and C44 reuse ids the same way. Read a shared row as the concept, not
as the syntax to grep for, and pull the language-specific prose when the row is
close but the syntax does not match.
| Code |
Scanner |
Severity |
What to look for |
| C1 |
py |
LOW |
Assertion inside conditional or loop that may never run |
| C2 |
py/js/rf |
HIGH |
Test body contains no assertion at all |
| C2b |
py/js/rf |
LOW |
Test calls production code but verifies nothing |
| C2c |
py |
LOW |
Empty self.subTest(...) block |
| C3 |
py/rf |
HIGH |
Assert inside try whose except swallows the error |
| C4 |
py |
HIGH |
Test function not collected by pytest |
| C4b |
py |
LOW |
Test class has __init__ (pytest won't collect it) |
| C5 |
py/js/rf |
HIGH |
Always-true assertion |
| C6 |
py/js/rf |
LOW |
Weak assertion: only checks that something came back |
| C6b |
py |
LOW |
Assertion on positional mock argument via computed index |
| C6c |
py |
LOW |
Mock call_count truthiness as the oracle |
| C7 |
py/js/rf |
HIGH |
Self-comparison: both sides are identical |
| C8 |
py/js |
LOW |
Float exact equality |
| C8b |
py/js |
LOW |
Approximate equality with no explicit tolerance |
| C9 |
py/js/rf |
LOW |
pytest.raises too broad |
| C9b |
rf |
- |
RequestsLibrary expected_status=any |
| C11a |
py/js/rf |
LOW |
Self-confirming literal: test assigns then asserts the same value |
| C13 |
py |
HIGH |
Mock assertion misspelled or not called |
| C13b |
py |
LOW |
patch() without autospec |
| C14 |
py |
LOW |
Golden file generated from the actual output |
| C16 |
py/js/rf |
LOW |
Result depends on uncontrolled time, randomness, or sleep |
| C17 |
py |
HIGH |
pytest.skip() inside broad except |
| C18 |
py/js |
LOW |
String/repr comparison |
| C19 |
py |
LOW |
pytest.raises wraps more than one call |
| C20 |
py/js/rf |
HIGH |
Assertion after unconditional return/raise/fail |
| C21 |
py/js/rf |
LOW |
Every assertion is inside a conditional; none runs unconditionally |
| C22 |
py |
OFF |
Async test never awaits the unit under test |
| C23 |
py/js/rf |
LOW |
Hard-coded absolute or home-relative file path |
| C24 |
py |
LOW |
Module-level mutable state mutated by test |
| C25 |
py |
LOW |
xfail without strict=True |
| C27 |
py |
HIGH |
try/except/pass around SUT call with no assertion |
| C28 |
py |
LOW |
pytest.raises binding variable never read |
| C29 |
py |
LOW |
os.environ modified directly in test |
| C30 |
py |
LOW |
HTTP mock not activated |
| C31 |
py/rf |
LOW |
capsys.readouterr() result discarded |
| C32 |
py/rf |
LOW |
@pytest.mark.skip without reason |
| C33 |
py |
LOW |
ML metric computed but not asserted |
| C34 |
py |
LOW |
Suboptimal assertion form |
| C35 |
py |
LOW |
Retry/flaky decorator |
| C36 |
py |
LOW |
pytest.fail() without reason |
| C37 |
py/js/rf |
LOW |
Duplicate parametrize case |
| C38 |
py |
HIGH |
Two tests share a name |
| C39 |
py |
HIGH |
Returns a comparison instead of asserting |
| C41 |
py |
LOW |
Assertion on a None-returning mutator |
| C42 |
py |
HIGH |
Assertion on a generator/lambda |
| C43 |
py |
LOW |
Mid-test skip |
| C44 |
py/js/rf |
HIGH |
Numeric tautology |
| C45 |
py |
HIGH |
Empty parametrize |
| C48 |
py/js |
LOW |
Dark patch: flips a test-mode flag then asserts |
| C49 |
py |
LOW |
pytest.warns/assertWarns wraps more than one call |
| C50 |
py |
LOW |
Captured log never asserted |
| C51 |
py |
HIGH |
Empty-bodied pytest.raises/warns context |
| C52 |
py |
LOW |
Membership self-confirmation |
| C55 |
py |
LOW |
Assertion compares two mock-rooted values |
| C56 |
py |
LOW |
Sync assert of a never-awaited coroutine |
| C57 |
py |
LOW |
Assertion against an unconfigured Mock attribute |
| C59 |
py |
HIGH |
Bare comparison written as a statement |
| CC |
py/js/rf |
LOW |
Commented-out assert |
| D1 |
py/js |
LOW |
Assertion Roulette: multiple asserts, none with a message |
| D2 |
rf |
- |
Control flow at test level |
| D3 |
py/js |
LOW |
Duplicate Assert: same assertion appears twice |
| D4 |
py/js |
LOW |
Unnamed parametrize cases |
| D5 |
py |
LOW |
Excessive inline setup |
| D6 |
py/js |
LOW |
Debug print in test |
| D7 |
js |
LOW |
Anonymous test: empty or missing description |
| D8 |
js |
LOW |
Magic number in an assertion |
| JS1 |
js |
HIGH |
focused test (it.only/fit) skips the rest of the suite |
| JS2 |
js |
HIGH |
expect(x) with no matcher |
| JS3 |
js |
LOW |
snapshot is the only assertion |
| JS4 |
js |
LOW |
skipped test (it.skip/xit/it.todo) |
| JS5 |
js |
LOW |
async query/event not awaited (findBy*/waitFor/user-event) |
| JS6 |
js |
HIGH |
empty describe/suite |
| JS7 |
js |
LOW |
assertion in a non-awaited setTimeout/then callback |
| JS8 |
js |
LOW |
mocks the unit under test and asserts it directly |
| JS9 |
js |
HIGH |
assertion in a dead literal branch (if(false)) |
| JS11 |
js |
LOW |
try/catch swallows the assertion |
| JS13 |
js |
LOW |
queryBy*/queryAllBy* query (returns null when absent) as a loose statement, never asserted - getBy*/getAllBy*/findBy*/findAllBy* throw on absence and ARE the assertion |
| JS15 |
js |
LOW |
comparison wrapped in a boolean (expect(a===b).toBe(true)) |
| JS17 |
js |
LOW |
commented-out test block (// it(...)) |
| JS18 |
js |
LOW |
done callback instead of async/await |
| JS21 |
js |
HIGH |
matcher referenced but never called (expect(x).toBe with no ()) |
| JS22 |
js |
HIGH |
empty it.each/test.each table |
| JS23 |
js |
HIGH |
expect.assertions(N) with fewer unconditional reachable expect() calls than N |
| JS24 |
js |
LOW |
Cypress cy.get/find/contains query statement with no .should/.and/.then assertion |
| JS25 |
js |
HIGH |
the only assertion sits inside an array-iterator callback (forEach/map/filter/some/every/flatMap) - runs zero times on an empty collection |
| JS26 |
js |
LOW |
fake timers installed but never advanced (runAllTimers/advanceTimersByTime/tick) - the scheduled callback never fires, so the assertion reads un-mutated state |
| JS27 |
js |
LOW |
toHaveBeenCalled* is the sole oracle on a locally-created double - verifies wiring, not behaviour |
| JS29 |
js |
LOW |
expect(...).resolves/.rejects chain is a bare statement, not awaited or returned - the test finishes green before the matcher settles |
| JS30 |
js |
HIGH |
literal-vs-literal assertion (expect(2).toBe(3), chai expect(x).to.equal(y)) - both operands are fixed at parse time |
| JS31 |
js |
LOW |
try/catch swallows a possible throw with no assertion on the exception - a unit that stops throwing still passes green |
| M2 |
py/js/rf |
LOW |
Long test method |
| PL1 |
py |
- |
Asserts stripped at runtime |
| PL2 |
py |
- |
Warnings not promoted |
| PL7 |
py/js |
- |
No coverage gate |
| PL8 |
py/js |
- |
Run stops early |
| PL9 |
rf |
- |
Skip-on-failure run option |
| PL10 |
js |
- |
passWithNoTests |
| R1 |
rf |
- |
Forced green |
| R2 |
rf |
- |
Hollow verifier keyword |
| R3 |
rf |
- |
Test Cases in a .resource |
| R4 |
rf |
- |
No Operation only |
| R5 |
rf |
- |
Empty [Template] |
| R6 |
rf |
- |
Should Be True on a string literal |
| R7 |
rf |
- |
Hollow [Template] keyword |
| R8 |
rf |
- |
Verification only in Setup |
| R8b |
rf |
- |
Verification only in Teardown |
Semantic cases
Every AI-only S-code, plus the five numbered semantic cases. Complete: this is
what Step 4 screens against.
| Case |
Judgment |
Severity |
Name |
Rule |
| 10 |
J3 |
HIGH |
Mocks the unit under test |
Patches/mocks the function being tested, then asserts on the mock's return value |
| 11 |
J2/J3 |
HIGH |
Asserts the value fed to the mock |
Stubs dependency to return X, then asserts result == X with no real logic in between |
| 12 |
J2 |
HIGH |
Re-implements the production formula |
Expected value computed with the same formula as the SUT; both sides agree on the same wrong answer |
| 15 |
J6 |
HIGH |
Passes only if another test ran first |
Reads shared mutable state written by a sibling test; fails when run alone |
| 18 |
J2 |
HIGH |
Expected value contradicts what the code should do |
Asserts a value the independent oracle says is wrong; requires cited oracle before reporting |
| S1 |
J4 |
- |
Intent mismatch |
The name or docstring claims to verify X, the assertion checks Y or a trivial property (test_applies_discount that only asserts the call did not raise) |
| S2 |
J4 |
- |
Irrelevant oracle |
The assertion checks a property unrelated to the behavior under test: a test of the computed total that only asserts the response is not null |
| S3 |
J2 |
- |
Plausible-but-wrong expected value |
The expected constant looks reasonable but contradicts the spec (off-by-one, wrong rounding, wrong sign); derive the correct value from the spec and compare |
| S4 |
J4 |
- |
Oracle cannot distinguish correct from a likely bug |
The assertion passes for the right output and for a plausible wrong one: len(result) == 3 when the suspected bug also yields three items |
| S5 |
J3 |
- |
Tests the framework, not the code |
The assertion exercises a language or library guarantee (a dict stores a key, the ORM returns what was just saved) instead of the code under test |
| S6 |
J4 |
- |
Happy-path only against a stated contract |
The spec or docstring promises error handling or boundaries, the test covers only the nominal path |
| S7 |
J2 |
- |
Expected lifted from the output |
The expected value was copied from a run of the current code (a pasted dict, a captured response), so the test can only confirm the code matches itself |
| S8 |
J3 |
- |
Mock return reaches the assertion through an indirection |
The stub's value flows through one or two trivial steps to the assertion, so the test still echoes the stub instead of verifying real behavior |
| S9 |
J2 |
- |
Self-fulfilling arrangement |
The test arranges the exact state it then asserts, with no transformation by the unit under test |
| S10 |
J4 |
- |
Asserts the log, not the effect |
The only check is that a message was logged, not the state change the message describes |
| S11 |
J4 |
- |
Negative-only assertion on a security filter |
A sanitizer, redactor, or auth test asserts only that the bad thing is absent ("password" not in response); it passes when the output is empty or dropped, so require a paired positive assertion |
| S12 |
J3 |
- |
Patches core logic instead of an external edge |
The test patches a private method or a direct collaborator on the class under test, so the assertion reads the stub, not the unit's own logic; patching a genuine external edge is legitimate |
| S13 |
J6 |
- |
Passes only via shared state a sibling set up |
The test relies on module-global, fixture, or hoisted state that another test or an import mutates, so it passes only in a given execution order |
| S14 |
J2 |
- |
Recorded model output as the oracle |
Asserts == against a snapshotted LLM/model result; green means the model still emits what it once emitted, not that the output is correct |
| S15 |
J6 |
LOW |
Hand-rolled retry/poll loop masking flakiness |
Wraps action+assertion in a retry/poll and passes if any attempt succeeds; only the swallow-and-pass form (a retry that re-raises on exhaustion is a sanctioned settle, not S15) |
| S16 |
J4 |
LOW |
Call-verification as the sole oracle |
The only check is that a collaborator was called (assert_called_once/toHaveBeenCalled), with no assertion on the unit's own return value or state |
| S17 |
J4 |
HIGH |
Exception-path oracle blindness |
pytest.raises(Exception)/expect(fn).toThrow() with no type or message on a documented error contract; goes green when the exception came from arrange (typo, missing import, None-deref) and the SUT never reached its raise |
| S18 |
J3 |
LOW |
Contract-impossible stub value |
A legitimate edge stub is configured to return a value the real collaborator can never emit (negative price, schema-violating row, None where non-null is guaranteed); the SUT handles an unreachable branch while the real defect goes untouched |
| S21 |
J2 |
LOW |
Self-judging LLM/agent assertion |
The oracle is a live model call (judge_llm(...) == "yes", embedding-similarity against a model-generated reference, agent grading its own transcript); circular, passes whenever the judge is wrong in the same direction as the SUT |
Look-alike exemptions for the semantic codes
Check these before reporting any S-code. They override the table above: a pattern
listed here is correct code.
Look-alikes - do NOT flag: a deliberately narrow unit test whose scope the spec confirms
(S6 needs a stated broader contract); a constant that the spec genuinely endorses (not S3);
a sanitizer test that already pairs the negative check with a positive one (not S11); a test
of a filter whose contract is to drop the input entirely - a blocklist sanitizer, a
guard that returns empty on a forbidden value, a redactor that suppresses the whole field -
where empty output is the correct behavior, so the negative-only assertion legitimately
passes and a positive "content survived" assertion would contradict the design (not S11); a
mock on a genuine external edge - DB, network, clock (not S12); a jest.spyOn(instance, 'methodA') / vi.spyOn that stubs a DIFFERENT method than the one under test, to isolate an orchestrator from a sibling sub-unit (the assertion is on the composed result, not the stub) - S12 fires only when the patched symbol is a method of the SUT instance itself or the assertion echoes the stub value (not S12); a constructor-injected or module-level collaborator mock - repository, db, auth, or HTTP client (a clean case-10 external edge, not S12); a stub-config call made on the very library under test - mockingoose, tinyspy, jsdom-testing-mocks - where the mocking library IS the SUT, so the stub setup is production code (not S5/S8/C11a); a test whose shared state is
reset by an autouse/beforeEach teardown (not S13); a structural or contract assertion on a
model output - valid JSON, required keys present, a cited source id matches, a refusal on a
banned prompt, a deterministic post-processing step - or a mocked/stubbed model whose return is
fixture data (not S14); a sanctioned async-settling wait - Robot Wait Until Keyword Succeeds,
Testing Library waitFor/findBy*, Playwright/Cypress auto-wait, await expect(...).toPass() -
that polls a real settle condition and still fails hard on timeout (not S15); a call-only
assertion where the interaction IS the contract - a fire-and-forget event, an audit-log or
telemetry write, a queue publish - or a toHaveBeenCalledWith/assert_called_once_with that
pins the specific arguments, or any call-verification paired with an assertion on the SUT's return value or state - S16 requires the call-verification to be the SOLE oracle (not S16); a pytest.raises(SpecificError, match=...) bound to the SUT line (not S17); a stub fed a value the collaborator's contract can actually return (not S18); a test under *.problem.* / *.solution.* / exercises/ / katas/ / playground/ - a teaching or TDD-spec fixture whose expected value is intentional (the exercise IS the spec), not a frozen bug (not case 18, not S3); a deterministic rubric, structural validator, or frozen human-labeled judge set rather than a live model verdict (not S21).
Structural code index (TS/JS, Robot, project layer)
Generated from schema/code-catalog.json, so it cannot drift from
reference.md. Severity - means the code carries no fixed severity.
| Code |
Severity |
What to look for |
| TypeScript / JavaScript |
|
24 codes |
| JS1 |
HIGH |
focused test (it.only/fit) skips the rest of the suite |
| JS2 |
HIGH |
expect(x) with no matcher |
| JS3 |
LOW |
snapshot is the only assertion |
| JS4 |
LOW |
skipped test (it.skip/xit/it.todo) |
| JS5 |
LOW |
async query/event not awaited (findBy*/waitFor/user-event) |
| JS6 |
HIGH |
empty describe/suite |
| JS7 |
LOW |
assertion in a non-awaited setTimeout/then callback |
| JS8 |
LOW |
mocks the unit under test and asserts it directly |
| JS9 |
HIGH |
assertion in a dead literal branch (if(false)) |
| JS11 |
LOW |
try/catch swallows the assertion |
| JS13 |
LOW |
queryBy*/queryAllBy* query (returns null when absent) as a loose statement, never asserted - getBy*/getAllBy*/findBy*/findAllBy* throw on absence and ARE the assertion |
| JS15 |
LOW |
comparison wrapped in a boolean (expect(a===b).toBe(true)) |
| JS17 |
LOW |
commented-out test block (// it(...)) |
| JS18 |
LOW |
done callback instead of async/await |
| JS21 |
HIGH |
matcher referenced but never called (expect(x).toBe with no ()) |
| JS22 |
HIGH |
empty it.each/test.each table |
| JS23 |
HIGH |
expect.assertions(N) with fewer unconditional reachable expect() calls than N |
| JS24 |
LOW |
Cypress cy.get/find/contains query statement with no .should/.and/.then assertion |
| JS25 |
HIGH |
the only assertion sits inside an array-iterator callback (forEach/map/filter/some/every/flatMap) - runs zero times on an empty collection |
| JS26 |
LOW |
fake timers installed but never advanced (runAllTimers/advanceTimersByTime/tick) - the scheduled callback never fires, so the assertion reads un-mutated state |
| JS27 |
LOW |
toHaveBeenCalled* is the sole oracle on a locally-created double - verifies wiring, not behaviour |
| JS29 |
LOW |
expect(...).resolves/.rejects chain is a bare statement, not awaited or returned - the test finishes green before the matcher settles |
| JS30 |
HIGH |
literal-vs-literal assertion (expect(2).toBe(3), chai expect(x).to.equal(y)) - both operands are fixed at parse time |
| JS31 |
LOW |
try/catch swallows a possible throw with no assertion on the exception - a unit that stops throwing still passes green |
| Robot Framework |
|
9 codes |
| R1 |
- |
Forced green |
| R2 |
- |
Hollow verifier keyword |
| R3 |
- |
Test Cases in a .resource |
| R4 |
- |
No Operation only |
| R5 |
- |
Empty [Template] |
| R6 |
- |
Should Be True on a string literal |
| R7 |
- |
Hollow [Template] keyword |
| R8 |
- |
Verification only in Setup |
| R8b |
- |
Verification only in Teardown |
| Project layer |
|
6 codes |
| PL1 |
- |
Asserts stripped at runtime |
| PL2 |
- |
Warnings not promoted |
| PL7 |
- |
No coverage gate |
| PL8 |
- |
Run stops early |
| PL9 |
- |
Skip-on-failure run option |
| PL10 |
- |
passWithNoTests |
Protocol in one paragraph
A test is useful only if it fails when the code breaks. Detect the language and
framework, classify the test intent (spec, characterization, regression,
behavior), then apply the six judgments: J1 does the assertion run, J2 is the
expected value from an independent oracle, J3 is the real unit under test, J4
does the assertion verify enough, J5 is the test coupled to implementation
internals, J6 does the test pass in isolation. For Python, also run the full
structural catalog (families A-E, the 56 emitted C-codes). Flag only the first failing
judgment per test.
Non-negotiable rules
- Never report case 18 (expected value contradicts the spec) without citing an
independent oracle, and always run the adversarial verify pass first.
- Precision over recall: a wrong HIGH finding is worse than a missed LOW one.
- Check the look-alike exemptions in
reference.md before flagging.
Output
Use the CASE / SUMMARY format defined in the root SKILL.md Step 6:
CASE {number} ({J-code}) - {HIGH|LOW} - {language} - {intent}
Test / Finding / Evidence / Oracle (case 18 only) / Fix hint
1---2name: falsegreen-skill3description: Analyze test files for false-positive smells, meaning tests that pass even when the code breaks. Use when the user asks to review tests for false positives, weak assertions, mock misuse, test smells, or asks whether a test can actually fail. Covers Python (pytest, unittest), TypeScript, JavaScript (Jest, Vitest, Mocha), and Robot Framework.4license: MIT5---67# falsegreen-skill: false-positive test detection89This skill is a thin entry point. Default path: judge from this file. It carries10the protocol summary, the structural code index, the S-series table, and the11look-alike exemptions, which covers a Mode A review of a TypeScript, JavaScript,12or Robot Framework test end to end.1314Open a plugin-root file only where this file sends you:1516- `../../reference.md` - the full definition or worked example of one specific17 code, after the index below has named it. Read the passage, not the section.18- `../../SKILL.md` - the long-form J1-J6 wording, the Python structural catalog,19 Mode B (authoring), or Mode C (AI-fix). At ~36 KiB on a ~32 KiB host this20 replaces the rest of your context, so it is a decision, not a warm-up read.2122Do not open either before reading this file through.2324**Mandatory for every language, Python included: load the semantic catalog.** The25S-series (S1-S18, S21) is language-agnostic and lives in `reference.md` under26`## Patterns only the semantic pass can catch (AI-only)`, above the per-language27sections. The root `SKILL.md` names only S3 and S17, so a run that loads a28language section alone never sees the rest. The complete table is in this file29under "Semantic cases", with the look-alike exemptions right after it. Use both:30the S-series without its exemptions produces false positives, which this skill31treats as worse than a miss. `../../reference.md` has the same codes as prose with32examples if a finding needs the long form.3334**Mandatory for any non-Python file (JavaScript, TypeScript, Robot Framework,35Gherkin, Tavern): work the structural code index below before you judge.** It36names every code all three scanners emit, so it is what stops a non-Python review37under-detecting. The root `SKILL.md` tables are complete only for Python; for every38other language they are a summary.3940Reach for the matching `reference.md` section only when the index has named a41candidate and you need its full definition, and then read that passage, not the42section. On a ~32 KiB host a whole section does not fit beside this file: the TS/JS43section alone is ~19 KiB.4445This file is the floor: the structural code index names every code all three46scanners emit, the semantic table carries a row per S-code, and the exemptions47follow it. `../../reference.md` is ~92 KiB, so an eager full read overruns the48budget and truncates mid-file without warning, which is why the order above is49index first and passage second. `../../fragments/precision-rules.md` carries the50precision rules if the host resolves plugin-root paths.5152## Structural code index (all three scanners)5354Generated from `schema/code-catalog.json` and `schema/scanner-codes.json`, so it55cannot drift. Scanner column: `py` falsegreen, `js` falsegreen-js, `rf`56falsegreen-robot. Severity `-` means no fixed severity. Diagnostic codes57(D-series, M2) are opt-in: apply them only when asked.5859One limit worth knowing before you match on a row. A code shared across languages60carries one title here, and for the C-series that title is the Python form. C31 is61the clearest case: the row reads "capsys.readouterr() result discarded", and the62Robot form of the same id is any captured keyword value that is never asserted on.63C3, C9, C23 and C44 reuse ids the same way. Read a shared row as the concept, not64as the syntax to grep for, and pull the language-specific prose when the row is65close but the syntax does not match.6667<!-- fg:structural-codes-all:start -->68| Code | Scanner | Severity | What to look for |69|---|---|---|---|70| C1 | py | LOW | Assertion inside conditional or loop that may never run |71| C2 | py/js/rf | HIGH | Test body contains no assertion at all |72| C2b | py/js/rf | LOW | Test calls production code but verifies nothing |73| C2c | py | LOW | Empty `self.subTest(...)` block |74| C3 | py/rf | HIGH | Assert inside try whose except swallows the error |75| C4 | py | HIGH | Test function not collected by pytest |76| C4b | py | LOW | Test class has `__init__` (pytest won't collect it) |77| C5 | py/js/rf | HIGH | Always-true assertion |78| C6 | py/js/rf | LOW | Weak assertion: only checks that something came back |79| C6b | py | LOW | Assertion on positional mock argument via computed index |80| C6c | py | LOW | Mock `call_count` truthiness as the oracle |81| C7 | py/js/rf | HIGH | Self-comparison: both sides are identical |82| C8 | py/js | LOW | Float exact equality |83| C8b | py/js | LOW | Approximate equality with no explicit tolerance |84| C9 | py/js/rf | LOW | pytest.raises too broad |85| C9b | rf | - | RequestsLibrary `expected_status=any` |86| C11a | py/js/rf | LOW | Self-confirming literal: test assigns then asserts the same value |87| C13 | py | HIGH | Mock assertion misspelled or not called |88| C13b | py | LOW | patch() without autospec |89| C14 | py | LOW | Golden file generated from the actual output |90| C16 | py/js/rf | LOW | Result depends on uncontrolled time, randomness, or sleep |91| C17 | py | HIGH | pytest.skip() inside broad except |92| C18 | py/js | LOW | String/repr comparison |93| C19 | py | LOW | pytest.raises wraps more than one call |94| C20 | py/js/rf | HIGH | Assertion after unconditional return/raise/fail |95| C21 | py/js/rf | LOW | Every assertion is inside a conditional; none runs unconditionally |96| C22 | py | OFF | Async test never awaits the unit under test |97| C23 | py/js/rf | LOW | Hard-coded absolute or home-relative file path |98| C24 | py | LOW | Module-level mutable state mutated by test |99| C25 | py | LOW | xfail without strict=True |100| C27 | py | HIGH | try/except/pass around SUT call with no assertion |101| C28 | py | LOW | pytest.raises binding variable never read |102| C29 | py | LOW | os.environ modified directly in test |103| C30 | py | LOW | HTTP mock not activated |104| C31 | py/rf | LOW | capsys.readouterr() result discarded |105| C32 | py/rf | LOW | @pytest.mark.skip without reason |106| C33 | py | LOW | ML metric computed but not asserted |107| C34 | py | LOW | Suboptimal assertion form |108| C35 | py | LOW | Retry/flaky decorator |109| C36 | py | LOW | pytest.fail() without reason |110| C37 | py/js/rf | LOW | Duplicate parametrize case |111| C38 | py | HIGH | Two tests share a name |112| C39 | py | HIGH | Returns a comparison instead of asserting |113| C41 | py | LOW | Assertion on a None-returning mutator |114| C42 | py | HIGH | Assertion on a generator/lambda |115| C43 | py | LOW | Mid-test skip |116| C44 | py/js/rf | HIGH | Numeric tautology |117| C45 | py | HIGH | Empty parametrize |118| C48 | py/js | LOW | Dark patch: flips a test-mode flag then asserts |119| C49 | py | LOW | `pytest.warns`/`assertWarns` wraps more than one call |120| C50 | py | LOW | Captured log never asserted |121| C51 | py | HIGH | Empty-bodied `pytest.raises`/`warns` context |122| C52 | py | LOW | Membership self-confirmation |123| C55 | py | LOW | Assertion compares two mock-rooted values |124| C56 | py | LOW | Sync assert of a never-awaited coroutine |125| C57 | py | LOW | Assertion against an unconfigured Mock attribute |126| C59 | py | HIGH | Bare comparison written as a statement |127| CC | py/js/rf | LOW | Commented-out assert |128| D1 | py/js | LOW | Assertion Roulette: multiple asserts, none with a message |129| D2 | rf | - | Control flow at test level |130| D3 | py/js | LOW | Duplicate Assert: same assertion appears twice |131| D4 | py/js | LOW | Unnamed parametrize cases |132| D5 | py | LOW | Excessive inline setup |133| D6 | py/js | LOW | Debug print in test |134| D7 | js | LOW | Anonymous test: empty or missing description |135| D8 | js | LOW | Magic number in an assertion |136| JS1 | js | HIGH | focused test (`it.only`/`fit`) skips the rest of the suite |137| JS2 | js | HIGH | `expect(x)` with no matcher |138| JS3 | js | LOW | snapshot is the only assertion |139| JS4 | js | LOW | skipped test (`it.skip`/`xit`/`it.todo`) |140| JS5 | js | LOW | async query/event not awaited (`findBy*`/`waitFor`/user-event) |141| JS6 | js | HIGH | empty `describe`/`suite` |142| JS7 | js | LOW | assertion in a non-awaited `setTimeout`/`then` callback |143| JS8 | js | LOW | mocks the unit under test and asserts it directly |144| JS9 | js | HIGH | assertion in a dead literal branch (`if(false)`) |145| JS11 | js | LOW | `try/catch` swallows the assertion |146| JS13 | js | LOW | `queryBy*`/`queryAllBy*` query (returns null when absent) as a loose statement, never asserted - `getBy*`/`getAllBy*`/`findBy*`/`findAllBy*` throw on absence and ARE the assertion |147| JS15 | js | LOW | comparison wrapped in a boolean (`expect(a===b).toBe(true)`) |148| JS17 | js | LOW | commented-out test block (`// it(...)`) |149| JS18 | js | LOW | `done` callback instead of async/await |150| JS21 | js | HIGH | matcher referenced but never called (`expect(x).toBe` with no `()`) |151| JS22 | js | HIGH | empty `it.each`/`test.each` table |152| JS23 | js | HIGH | `expect.assertions(N)` with fewer unconditional reachable `expect()` calls than `N` |153| JS24 | js | LOW | Cypress `cy.get/find/contains` query statement with no `.should`/`.and`/`.then` assertion |154| JS25 | js | HIGH | the only assertion sits inside an array-iterator callback (`forEach`/`map`/`filter`/`some`/`every`/`flatMap`) - runs zero times on an empty collection |155| JS26 | js | LOW | fake timers installed but never advanced (`runAllTimers`/`advanceTimersByTime`/`tick`) - the scheduled callback never fires, so the assertion reads un-mutated state |156| JS27 | js | LOW | `toHaveBeenCalled*` is the sole oracle on a locally-created double - verifies wiring, not behaviour |157| JS29 | js | LOW | `expect(...).resolves`/`.rejects` chain is a bare statement, not awaited or returned - the test finishes green before the matcher settles |158| JS30 | js | HIGH | literal-vs-literal assertion (`expect(2).toBe(3)`, chai `expect(x).to.equal(y)`) - both operands are fixed at parse time |159| JS31 | js | LOW | `try/catch` swallows a possible throw with no assertion on the exception - a unit that stops throwing still passes green |160| M2 | py/js/rf | LOW | Long test method |161| PL1 | py | - | Asserts stripped at runtime |162| PL2 | py | - | Warnings not promoted |163| PL7 | py/js | - | No coverage gate |164| PL8 | py/js | - | Run stops early |165| PL9 | rf | - | Skip-on-failure run option |166| PL10 | js | - | passWithNoTests |167| R1 | rf | - | Forced green |168| R2 | rf | - | Hollow verifier keyword |169| R3 | rf | - | Test Cases in a .resource |170| R4 | rf | - | No Operation only |171| R5 | rf | - | Empty [Template] |172| R6 | rf | - | Should Be True on a string literal |173| R7 | rf | - | Hollow [Template] keyword |174| R8 | rf | - | Verification only in Setup |175| R8b | rf | - | Verification only in Teardown |176<!-- fg:structural-codes-all:end -->177178## Semantic cases179180Every AI-only S-code, plus the five numbered semantic cases. Complete: this is181what Step 4 screens against.182183<!-- fg:semantic-cases-compact:start -->184| Case | Judgment | Severity | Name | Rule |185|---|---|---|---|---|186| 10 | J3 | HIGH | Mocks the unit under test | Patches/mocks the function being tested, then asserts on the mock's return value |187| 11 | J2/J3 | HIGH | Asserts the value fed to the mock | Stubs dependency to return X, then asserts result == X with no real logic in between |188| 12 | J2 | HIGH | Re-implements the production formula | Expected value computed with the same formula as the SUT; both sides agree on the same wrong answer |189| 15 | J6 | HIGH | Passes only if another test ran first | Reads shared mutable state written by a sibling test; fails when run alone |190| 18 | J2 | HIGH | Expected value contradicts what the code should do | Asserts a value the independent oracle says is wrong; requires cited oracle before reporting |191| S1 | J4 | - | Intent mismatch | The name or docstring claims to verify X, the assertion checks Y or a trivial property (`test_applies_discount` that only asserts the call did not raise) |192| S2 | J4 | - | Irrelevant oracle | The assertion checks a property unrelated to the behavior under test: a test of the computed total that only asserts the response is not null |193| S3 | J2 | - | Plausible-but-wrong expected value | The expected constant looks reasonable but contradicts the spec (off-by-one, wrong rounding, wrong sign); derive the correct value from the spec and compare |194| S4 | J4 | - | Oracle cannot distinguish correct from a likely bug | The assertion passes for the right output and for a plausible wrong one: `len(result) == 3` when the suspected bug also yields three items |195| S5 | J3 | - | Tests the framework, not the code | The assertion exercises a language or library guarantee (a dict stores a key, the ORM returns what was just saved) instead of the code under test |196| S6 | J4 | - | Happy-path only against a stated contract | The spec or docstring promises error handling or boundaries, the test covers only the nominal path |197| S7 | J2 | - | Expected lifted from the output | The expected value was copied from a run of the current code (a pasted dict, a captured response), so the test can only confirm the code matches itself |198| S8 | J3 | - | Mock return reaches the assertion through an indirection | The stub's value flows through one or two trivial steps to the assertion, so the test still echoes the stub instead of verifying real behavior |199| S9 | J2 | - | Self-fulfilling arrangement | The test arranges the exact state it then asserts, with no transformation by the unit under test |200| S10 | J4 | - | Asserts the log, not the effect | The only check is that a message was logged, not the state change the message describes |201| S11 | J4 | - | Negative-only assertion on a security filter | A sanitizer, redactor, or auth test asserts only that the bad thing is absent (`"password" not in response`); it passes when the output is empty or dropped, so require a paired positive assertion |202| S12 | J3 | - | Patches core logic instead of an external edge | The test patches a private method or a direct collaborator on the class under test, so the assertion reads the stub, not the unit's own logic; patching a genuine external edge is legitimate |203| S13 | J6 | - | Passes only via shared state a sibling set up | The test relies on module-global, fixture, or hoisted state that another test or an import mutates, so it passes only in a given execution order |204| S14 | J2 | - | Recorded model output as the oracle | Asserts `==` against a snapshotted LLM/model result; green means the model still emits what it once emitted, not that the output is correct |205| S15 | J6 | LOW | Hand-rolled retry/poll loop masking flakiness | Wraps action+assertion in a retry/poll and passes if any attempt succeeds; only the swallow-and-pass form (a retry that re-raises on exhaustion is a sanctioned settle, not S15) |206| S16 | J4 | LOW | Call-verification as the sole oracle | The only check is that a collaborator was called (`assert_called_once`/`toHaveBeenCalled`), with no assertion on the unit's own return value or state |207| S17 | J4 | HIGH | Exception-path oracle blindness | `pytest.raises(Exception)`/`expect(fn).toThrow()` with no type or message on a documented error contract; goes green when the exception came from arrange (typo, missing import, None-deref) and the SUT never reached its raise |208| S18 | J3 | LOW | Contract-impossible stub value | A legitimate edge stub is configured to return a value the real collaborator can never emit (negative price, schema-violating row, `None` where non-null is guaranteed); the SUT handles an unreachable branch while the real defect goes untouched |209| S21 | J2 | LOW | Self-judging LLM/agent assertion | The oracle is a live model call (`judge_llm(...) == "yes"`, embedding-similarity against a model-generated reference, agent grading its own transcript); circular, passes whenever the judge is wrong in the same direction as the SUT |210<!-- fg:semantic-cases-compact:end -->211212### Look-alike exemptions for the semantic codes213214Check these before reporting any S-code. They override the table above: a pattern215listed here is correct code.216217<!-- fg:semantic-exemptions:start -->218Look-alikes - do NOT flag: a deliberately narrow unit test whose scope the spec confirms219(S6 needs a stated broader contract); a constant that the spec genuinely endorses (not S3);220a sanitizer test that already pairs the negative check with a positive one (not S11); a test221of a filter whose contract is to drop the input entirely - a blocklist sanitizer, a222guard that returns empty on a forbidden value, a redactor that suppresses the whole field -223where empty output is the correct behavior, so the negative-only assertion legitimately224passes and a positive "content survived" assertion would contradict the design (not S11); a225mock on a genuine external edge - DB, network, clock (not S12); a `jest.spyOn(instance, 'methodA')` / `vi.spyOn` that stubs a DIFFERENT method than the one under test, to isolate an orchestrator from a sibling sub-unit (the assertion is on the composed result, not the stub) - S12 fires only when the patched symbol is a method of the SUT instance itself or the assertion echoes the stub value (not S12); a constructor-injected or module-level collaborator mock - repository, db, auth, or HTTP client (a clean case-10 external edge, not S12); a stub-config call made on the very library under test - `mockingoose`, `tinyspy`, `jsdom-testing-mocks` - where the mocking library IS the SUT, so the stub setup is production code (not S5/S8/C11a); a test whose shared state is226reset by an autouse/`beforeEach` teardown (not S13); a structural or contract assertion on a227model output - valid JSON, required keys present, a cited source id matches, a refusal on a228banned prompt, a deterministic post-processing step - or a mocked/stubbed model whose return is229fixture data (not S14); a sanctioned async-settling wait - Robot `Wait Until Keyword Succeeds`,230Testing Library `waitFor`/`findBy*`, Playwright/Cypress auto-wait, `await expect(...).toPass()` -231that polls a real settle condition and still fails hard on timeout (not S15); a call-only232assertion where the interaction IS the contract - a fire-and-forget event, an audit-log or233telemetry write, a queue publish - or a `toHaveBeenCalledWith`/`assert_called_once_with` that234pins the specific arguments, or any call-verification paired with an assertion on the SUT's return value or state - S16 requires the call-verification to be the SOLE oracle (not S16); a `pytest.raises(SpecificError, match=...)` bound to the SUT line (not S17); a stub fed a value the collaborator's contract can actually return (not S18); a test under `*.problem.*` / `*.solution.*` / `exercises/` / `katas/` / `playground/` - a teaching or TDD-spec fixture whose expected value is intentional (the exercise IS the spec), not a frozen bug (not case 18, not S3); a deterministic rubric, structural validator, or frozen human-labeled judge set rather than a live model verdict (not S21).235<!-- fg:semantic-exemptions:end -->236237## Structural code index (TS/JS, Robot, project layer)238239Generated from `schema/code-catalog.json`, so it cannot drift from240`reference.md`. Severity `-` means the code carries no fixed severity.241242<!-- fg:structural-codes-compact:start -->243| Code | Severity | What to look for |244|---|---|---|245| **TypeScript / JavaScript** | | 24 codes |246| JS1 | HIGH | focused test (`it.only`/`fit`) skips the rest of the suite |247| JS2 | HIGH | `expect(x)` with no matcher |248| JS3 | LOW | snapshot is the only assertion |249| JS4 | LOW | skipped test (`it.skip`/`xit`/`it.todo`) |250| JS5 | LOW | async query/event not awaited (`findBy*`/`waitFor`/user-event) |251| JS6 | HIGH | empty `describe`/`suite` |252| JS7 | LOW | assertion in a non-awaited `setTimeout`/`then` callback |253| JS8 | LOW | mocks the unit under test and asserts it directly |254| JS9 | HIGH | assertion in a dead literal branch (`if(false)`) |255| JS11 | LOW | `try/catch` swallows the assertion |256| JS13 | LOW | `queryBy*`/`queryAllBy*` query (returns null when absent) as a loose statement, never asserted - `getBy*`/`getAllBy*`/`findBy*`/`findAllBy*` throw on absence and ARE the assertion |257| JS15 | LOW | comparison wrapped in a boolean (`expect(a===b).toBe(true)`) |258| JS17 | LOW | commented-out test block (`// it(...)`) |259| JS18 | LOW | `done` callback instead of async/await |260| JS21 | HIGH | matcher referenced but never called (`expect(x).toBe` with no `()`) |261| JS22 | HIGH | empty `it.each`/`test.each` table |262| JS23 | HIGH | `expect.assertions(N)` with fewer unconditional reachable `expect()` calls than `N` |263| JS24 | LOW | Cypress `cy.get/find/contains` query statement with no `.should`/`.and`/`.then` assertion |264| JS25 | HIGH | the only assertion sits inside an array-iterator callback (`forEach`/`map`/`filter`/`some`/`every`/`flatMap`) - runs zero times on an empty collection |265| JS26 | LOW | fake timers installed but never advanced (`runAllTimers`/`advanceTimersByTime`/`tick`) - the scheduled callback never fires, so the assertion reads un-mutated state |266| JS27 | LOW | `toHaveBeenCalled*` is the sole oracle on a locally-created double - verifies wiring, not behaviour |267| JS29 | LOW | `expect(...).resolves`/`.rejects` chain is a bare statement, not awaited or returned - the test finishes green before the matcher settles |268| JS30 | HIGH | literal-vs-literal assertion (`expect(2).toBe(3)`, chai `expect(x).to.equal(y)`) - both operands are fixed at parse time |269| JS31 | LOW | `try/catch` swallows a possible throw with no assertion on the exception - a unit that stops throwing still passes green |270| **Robot Framework** | | 9 codes |271| R1 | - | Forced green |272| R2 | - | Hollow verifier keyword |273| R3 | - | Test Cases in a .resource |274| R4 | - | No Operation only |275| R5 | - | Empty [Template] |276| R6 | - | Should Be True on a string literal |277| R7 | - | Hollow [Template] keyword |278| R8 | - | Verification only in Setup |279| R8b | - | Verification only in Teardown |280| **Project layer** | | 6 codes |281| PL1 | - | Asserts stripped at runtime |282| PL2 | - | Warnings not promoted |283| PL7 | - | No coverage gate |284| PL8 | - | Run stops early |285| PL9 | - | Skip-on-failure run option |286| PL10 | - | passWithNoTests |287<!-- fg:structural-codes-compact:end -->288289## Protocol in one paragraph290291A test is useful only if it fails when the code breaks. Detect the language and292framework, classify the test intent (spec, characterization, regression,293behavior), then apply the six judgments: J1 does the assertion run, J2 is the294expected value from an independent oracle, J3 is the real unit under test, J4295does the assertion verify enough, J5 is the test coupled to implementation296internals, J6 does the test pass in isolation. For Python, also run the full297structural catalog (families A-E, the 56 emitted C-codes). Flag only the first failing298judgment per test.299300## Non-negotiable rules301302- Never report case 18 (expected value contradicts the spec) without citing an303 independent oracle, and always run the adversarial verify pass first.304- Precision over recall: a wrong HIGH finding is worse than a missed LOW one.305- Check the look-alike exemptions in `reference.md` before flagging.306307## Output308309Use the CASE / SUMMARY format defined in the root `SKILL.md` Step 6:310311```312CASE {number} ({J-code}) - {HIGH|LOW} - {language} - {intent}313Test / Finding / Evidence / Oracle (case 18 only) / Fix hint314```