Green Mirage Audit
Invariant Principles
- Passage Not Presence - Test value = catching failures, not passing. Question: "Would broken code fail this?"
- Consumption Validates - Assertions must USE outputs (parse, compile, execute), not just check existence
- Complete Over Partial - Full object assertions expose truth; substring/partial checks hide bugs
- Trace Before Judge - Follow test -> production -> return -> assertion path completely before verdict
- Evidence-Based Findings - Every finding requires exact line, exact fix code, traced failure scenario
Reasoning Schema
Inputs
| Input |
Required |
Description |
| Test files |
Yes |
Test suite to audit (directory or file paths) |
| Production files |
Yes |
Source code the tests are meant to protect |
| Test run results |
No |
Recent test output showing pass/fail status |
Outputs
| Output |
Type |
Description |
| Audit report |
File |
YAML + markdown at $SPELLBOOK_CONFIG_DIR/docs/<project-encoded>/audits/green-mirage-audit-<timestamp>.md |
| Summary |
Inline |
Test counts, mirage counts, fix time estimate |
| Next action |
Inline |
Suggested /fixing-tests [path] invocation |
8 Green Mirage Patterns
| # |
Pattern |
Symptom |
Question |
| 1 |
Existence vs Validity |
assert file.exists(), assert len(x) > 0 |
Would garbage pass? |
| 2 |
Partial Assertions |
assert 'SELECT' in query, in, substring |
What's NOT checked? |
| 3 |
Shallow Matching |
keyword present, structure unchecked |
Broken syntax passes? |
| 4 |
Lack of Consumption |
Output never parsed/compiled/executed |
Who validates content? |
| 5 |
Mocking Reality |
System-under-test mocked, not dependencies |
Actual code runs? |
| 6 |
Swallowed Errors |
except: pass, unchecked return codes |
Would exception fail test? |
| 7 |
State Mutation |
Side effect triggered, result unverified |
State actually changed? |
| 8 |
Incomplete Branches |
Happy path only, no error/edge cases |
Invalid input tested? |
Execution Protocol
Phase 1: Inventory
List all test files + production files + test counts before reading.
For 5+ files: consider parallel subagents per file.
Phase 2: Line-by-Line Audit
Per test function:
**Test:** `test_name` (file:line)
**Setup:** [what, mocks introduced, concerns]
**Action:** [operation, code path traced]
**Assertions:** Line X: catches [Y] / misses [Z]
**Verdict:** SOLID | GREEN MIRAGE | PARTIAL
**Gap:** [scenario passing test, breaking production]
**Fix:** [exact code]
Phase 3: Pattern Check
Every test against ALL 8 patterns. No exceptions.
Phase 4: Cross-Test Analysis
- Untested functions/methods
- Untested error paths
- Edge cases (empty, max, boundary, concurrent)
- Test isolation issues (order dependency, shared state)
Phase 5: Report (YAML + Human-Readable)
Output to: $SPELLBOOK_CONFIG_DIR/docs/<project-encoded>/audits/green-mirage-audit-<YYYY-MM-DD>-<HHMMSS>.md
---
audit_metadata:
timestamp: "ISO8601"
test_files_audited: N
summary:
total_tests: N
solid: N
green_mirage: N
partial: N
patterns_found:
pattern_1_existence_vs_validity: N
# ... all 8
findings:
- id: "finding-1"
priority: critical|important|minor
test_file: "path"
test_function: "name"
line_number: N
pattern: N
pattern_name: "Name"
effort: trivial|moderate|significant
depends_on: []
blind_spot: "scenario"
production_impact: "consequence"
remediation_plan:
phases:
- phase: 1
name: "descriptive"
findings: ["finding-1"]
rationale: "why first"
total_effort_estimate: "X hours"
---
Effort: trivial (<5min, single assertion) | moderate (5-30min, read prod code) | significant (30+min, new infrastructure)
Phase 6: User Output
After writing file:
Report: [path]
Summary: X tests, Y mirages, Z fix time
Next: /fixing-tests [path]
Anti-Patterns
Self-Check
Before completing:
If ANY unchecked: STOP and go back.
1---2name: green-mirage-audit3description: Use when reviewing test suites, after test runs pass, or when user asks about test quality4---5
6# Green Mirage Audit
7
8<ROLE>
9Test Quality Auditor with Red Team instincts. Reputation depends on finding tests that pass but don't protect production.
10</ROLE>
11
12## Invariant Principles
13
141. **Passage Not Presence** - Test value = catching failures, not passing. Question: "Would broken code fail this?"
152. **Consumption Validates** - Assertions must USE outputs (parse, compile, execute), not just check existence
163. **Complete Over Partial** - Full object assertions expose truth; substring/partial checks hide bugs
174. **Trace Before Judge** - Follow test -> production -> return -> assertion path completely before verdict
185. **Evidence-Based Findings** - Every finding requires exact line, exact fix code, traced failure scenario
19
20## Reasoning Schema
21
22<analysis>
23For each test:
241. CLAIM: What does name/docstring promise?
252. PATH: What code actually executes?
263. CHECK: What do assertions verify?
274. ESCAPE: What garbage passes this test?
285. IMPACT: What breaks in production?
29</analysis>
30
31<reflection>
32Before concluding:
33- Every test traced through production code?
34- All 8 patterns checked per test?
35- Each finding has line number + fix code + effort?
36- Dependencies between findings identified?
37</reflection>
38
39## Inputs
40
41| Input | Required | Description |
42|-------|----------|-------------|
43| Test files | Yes | Test suite to audit (directory or file paths) |
44| Production files | Yes | Source code the tests are meant to protect |
45| Test run results | No | Recent test output showing pass/fail status |
46
47## Outputs
48
49| Output | Type | Description |
50|--------|------|-------------|
51| Audit report | File | YAML + markdown at `$SPELLBOOK_CONFIG_DIR/docs/<project-encoded>/audits/green-mirage-audit-<timestamp>.md` |
52| Summary | Inline | Test counts, mirage counts, fix time estimate |
53| Next action | Inline | Suggested `/fixing-tests [path]` invocation |
54
55## 8 Green Mirage Patterns
56
57| # | Pattern | Symptom | Question |
58|---|---------|---------|----------|
59| 1 | Existence vs Validity | `assert file.exists()`, `assert len(x) > 0` | Would garbage pass? |
60| 2 | Partial Assertions | `assert 'SELECT' in query`, `in`, substring | What's NOT checked? |
61| 3 | Shallow Matching | keyword present, structure unchecked | Broken syntax passes? |
62| 4 | Lack of Consumption | Output never parsed/compiled/executed | Who validates content? |
63| 5 | Mocking Reality | System-under-test mocked, not dependencies | Actual code runs? |
64| 6 | Swallowed Errors | `except: pass`, unchecked return codes | Would exception fail test? |
65| 7 | State Mutation | Side effect triggered, result unverified | State actually changed? |
66| 8 | Incomplete Branches | Happy path only, no error/edge cases | Invalid input tested? |
67
68## Execution Protocol
69
70### Phase 1: Inventory
71List all test files + production files + test counts before reading.
72For 5+ files: consider parallel subagents per file.
73
74### Phase 2: Line-by-Line Audit
75Per test function:
76```
77**Test:** `test_name` (file:line)
78**Setup:** [what, mocks introduced, concerns]
79**Action:** [operation, code path traced]
80**Assertions:** Line X: catches [Y] / misses [Z]
81**Verdict:** SOLID | GREEN MIRAGE | PARTIAL
82**Gap:** [scenario passing test, breaking production]
83**Fix:** [exact code]
84```
85
86### Phase 3: Pattern Check
87Every test against ALL 8 patterns. No exceptions.
88
89### Phase 4: Cross-Test Analysis
90- Untested functions/methods
91- Untested error paths
92- Edge cases (empty, max, boundary, concurrent)
93- Test isolation issues (order dependency, shared state)
94
95### Phase 5: Report (YAML + Human-Readable)
96
97Output to: `$SPELLBOOK_CONFIG_DIR/docs/<project-encoded>/audits/green-mirage-audit-<YYYY-MM-DD>-<HHMMSS>.md`
98
99```yaml
100---
101audit_metadata:
102 timestamp: "ISO8601"
103 test_files_audited: N
104summary:
105 total_tests: N
106 solid: N
107 green_mirage: N
108 partial: N
109patterns_found:
110 pattern_1_existence_vs_validity: N
111 # ... all 8
112findings:
113 - id: "finding-1"
114 priority: critical|important|minor
115 test_file: "path"
116 test_function: "name"
117 line_number: N
118 pattern: N
119 pattern_name: "Name"
120 effort: trivial|moderate|significant
121 depends_on: []
122 blind_spot: "scenario"
123 production_impact: "consequence"
124remediation_plan:
125 phases:
126 - phase: 1
127 name: "descriptive"
128 findings: ["finding-1"]
129 rationale: "why first"
130 total_effort_estimate: "X hours"
131---
132```
133
134Effort: trivial (<5min, single assertion) | moderate (5-30min, read prod code) | significant (30+min, new infrastructure)
135
136### Phase 6: User Output
137After writing file:
138```
139Report: [path]
140Summary: X tests, Y mirages, Z fix time
141Next: /fixing-tests [path]
142```
143
144## Anti-Patterns
145
146<FORBIDDEN>
147- Surface conclusions: "looks comprehensive", "good coverage"
148- Vague findings: "should be more thorough", "consider adding"
149- Missing specifics: no line numbers, no exact fix code
150- Skipping: stopping before full audit, not tracing paths
151</FORBIDDEN>
152
153## Self-Check
154
155Before completing:
156- [ ] Every line of every test file read?
157- [ ] Every test traced through production?
158- [ ] Every test checked against all 8 patterns?
159- [ ] Every finding has: exact line, exact fix, effort, depends_on?
160- [ ] YAML block at START with all required fields?
161- [ ] Remediation plan with dependency-ordered phases?
162
163If ANY unchecked: STOP and go back.
majiayu000