AI Output Review
Purpose
Act as a skeptical reviewer of AI-generated output before it is trusted or shipped. Catch the specific failure modes AI agents produce, and separate what is verified from what is asserted. Complements code-review, security-review, and final-quality-audit.
When to Use
- After any AI agent produces a plan, code change, document, or decision.
- Before acting on AI output that affects architecture, security, or scope.
- As an independent-review step in multi-agent runs (
../../system/MULTI_AGENT_RULES.md).
Inputs
- The AI-generated output under review.
- The original request, requirements, and approved plan/architecture.
Discovery Questions
- What was actually requested vs what the output delivered?
- Which claims are backed by evidence vs asserted?
- Does the output contradict the approved architecture or prior decisions?
Responsibilities
Evaluate the output for each failure mode:
- Unsupported assumptions — claims presented as fact without basis.
- Invented facts — fabricated APIs, files, paths, results, or citations.
- Conflicting architecture — contradicts the approved design or itself.
- Unnecessary dependencies — libraries added without a justified need.
- Security risks — introduced or ignored (hand depth to
security-review).
- Missing tests — changes without corresponding verification.
- Over-engineering — complexity beyond the requirement.
- Scope expansion — work beyond what was approved.
- Outdated patterns — deprecated/legacy approaches where current ones exist.
- Incomplete validation — checks claimed but not actually run.
- False claims of completion — "done"/"working" without evidence.
Report each hit with location, why it's a problem, and the correction.
Required Workflow
- Read the request/plan and the AI output side by side.
- Walk the 11 failure modes.
- For each finding: cite location, classify the failure mode, state the fix.
- Separate verified from asserted claims.
- Issue a verdict: accept / revise / reject, with blockers.
Decision Rules
- Treat any "done/working/verified" without evidence as a false-completion finding until proven.
- Fabricated file/API/result → reject that portion outright.
- Architecture conflict or scope expansion → block until reconciled with the approved plan.
- Route deep security/perf concerns to
security-review / performance-review.
Rules
- Be skeptical by default; assertion is not evidence.
- Cite specifics; no vague "looks off."
- Don't approve output you can't verify — mark it "unverified."
Anti-Patterns
- Trusting confident phrasing as correctness.
- Passing output with fabricated facts or unrun "validation."
- Missing silent scope expansion because the result "looks good."
- Reviewing only style while ignoring invented facts.
Validation Checklist
Definition of Done
A review report covering all 11 failure modes with located findings and fixes, a clean separation of verified vs asserted, and an accept/revise/reject verdict — no fabricated or unvalidated output passed as done.
Related Skills
code-review, security-review, performance-review, final-quality-audit, project-orchestrator, documentation.
Related Knowledge
../../projects/current/ (approved plan/architecture to check against).
Related References
../../checklists/ (review checklists).
Context Loading Guidance
- Requires: the AI output, the original request, and the approved plan/architecture.
- Does not require: the whole repo (unless verifying a specific claim), unrelated references.
- May load:
security-review/performance-review for deep concerns; the specific file to verify an invented-fact claim.
- Stop when: the verdict and findings are delivered.
Token Efficiency Guidance
Compare output against the recorded plan, not the entire history. Verify suspicious claims by checking the specific file/command rather than reloading everything.
1---2name: ai-output-review3description: Use to critically review AI-generated output (plans, code, docs, decisions) for the failure modes AI agents are prone to — unsupported assumptions, invented facts, conflicting architecture, unnecessary dependencies, security risks, missing tests, over-engineering, scope expansion, outdated patterns, incomplete validation, and false claims of completion.4---56# AI Output Review78## Purpose910Act as a skeptical reviewer of AI-generated output before it is trusted or shipped. Catch the specific failure modes AI agents produce, and separate what is verified from what is asserted. Complements `code-review`, `security-review`, and `final-quality-audit`.1112## When to Use1314- After any AI agent produces a plan, code change, document, or decision.15- Before acting on AI output that affects architecture, security, or scope.16- As an independent-review step in multi-agent runs (`../../system/MULTI_AGENT_RULES.md`).1718## Inputs1920- The AI-generated output under review.21- The original request, requirements, and approved plan/architecture.2223## Discovery Questions2425- What was actually requested vs what the output delivered?26- Which claims are backed by evidence vs asserted?27- Does the output contradict the approved architecture or prior decisions?2829## Responsibilities3031Evaluate the output for each failure mode:321. **Unsupported assumptions** — claims presented as fact without basis.332. **Invented facts** — fabricated APIs, files, paths, results, or citations.343. **Conflicting architecture** — contradicts the approved design or itself.354. **Unnecessary dependencies** — libraries added without a justified need.365. **Security risks** — introduced or ignored (hand depth to `security-review`).376. **Missing tests** — changes without corresponding verification.387. **Over-engineering** — complexity beyond the requirement.398. **Scope expansion** — work beyond what was approved.409. **Outdated patterns** — deprecated/legacy approaches where current ones exist.4110. **Incomplete validation** — checks claimed but not actually run.4211. **False claims of completion** — "done"/"working" without evidence.4344Report each hit with location, why it's a problem, and the correction.4546## Required Workflow47481. Read the request/plan and the AI output side by side.492. Walk the 11 failure modes.503. For each finding: cite location, classify the failure mode, state the fix.514. Separate **verified** from **asserted** claims.525. Issue a verdict: accept / revise / reject, with blockers.5354## Decision Rules5556- Treat any "done/working/verified" without evidence as a **false-completion** finding until proven.57- Fabricated file/API/result → reject that portion outright.58- Architecture conflict or scope expansion → block until reconciled with the approved plan.59- Route deep security/perf concerns to `security-review` / `performance-review`.6061## Rules6263- Be skeptical by default; assertion is not evidence.64- Cite specifics; no vague "looks off."65- Don't approve output you can't verify — mark it "unverified."6667## Anti-Patterns6869- Trusting confident phrasing as correctness.70- Passing output with fabricated facts or unrun "validation."71- Missing silent scope expansion because the result "looks good."72- Reviewing only style while ignoring invented facts.7374## Validation Checklist7576- [ ] All 11 failure modes evaluated.77- [ ] Findings cite location + failure mode + fix.78- [ ] Verified vs asserted separated.79- [ ] Dependency additions justified or flagged.80- [ ] Completion claims backed by evidence or flagged false.81- [ ] Verdict (accept/revise/reject) with blockers.8283## Definition of Done8485A review report covering all 11 failure modes with located findings and fixes, a clean separation of verified vs asserted, and an accept/revise/reject verdict — no fabricated or unvalidated output passed as done.8687## Related Skills8889`code-review`, `security-review`, `performance-review`, `final-quality-audit`, `project-orchestrator`, `documentation`.9091## Related Knowledge9293`../../projects/current/` (approved plan/architecture to check against).9495## Related References9697`../../checklists/` (review checklists).9899## Context Loading Guidance100101- **Requires:** the AI output, the original request, and the approved plan/architecture.102- **Does not require:** the whole repo (unless verifying a specific claim), unrelated references.103- **May load:** `security-review`/`performance-review` for deep concerns; the specific file to verify an invented-fact claim.104- **Stop when:** the verdict and findings are delivered.105106## Token Efficiency Guidance107108Compare output against the recorded plan, not the entire history. Verify suspicious claims by checking the specific file/command rather than reloading everything.
ahtishamshahzad