Evaluate Plan

Evaluate Plan

Systematically verify that an AI-generated implementation plan fully covers its source requirements.

Context: First-pass plans silently drop ~20-40% of explicit requirements. A structured coverage check recovers most gaps. This skill forces a coverage-first second pass.

Phase 1: Identify the Source of Truth

Locate the document(s) the plan was built from:

• A PRD, spec, feature brief, or user story set
• An intent document or voice transcript
• The original user request in this conversation
• A file path provided as argument (e.g., evaluate plan against @prd.md)

If multiple sources exist, treat them as a combined requirement set.

If no explicit document exists: reconstruct the requirement set from the user's messages in the current conversation. Quote the original requests.

Rule: The source document is the authority. If something is in the source but not the plan, it is a gap — regardless of whether the plan's approach seems reasonable without it.

Phase 2: Extract Requirements Inventory

Read the source and extract every discrete requirement into a flat checklist:

Extraction rules:

• One requirement per line — do not bundle related items
• Preserve original language where possible
• Feelings and experiences are requirements
• Implicit requirements count (e.g., "like Spotlight" implies global hotkey, float-above, escape-to-dismiss)
• Do not merge for convenience — "user management" is not one requirement; break it into registration, login, password reset, roles, etc.

Phase 3: Evaluate Coverage

For each requirement, search the plan and assign a status:

Phase 4: Coverage Report

4a. Summary

COVERAGE REPORT
===============
Source: [document name(s) or "conversation context"]
Plan:   [plan name or "current plan in conversation"]
Total requirements: [N]

  ✅ Covered:         [n] ([%])
  🟡 Partial:         [n] ([%])
  ❌ Missing:          [n] ([%])
  ⚠️ Misinterpreted:   [n] ([%])

Coverage Score: [X]%

Score formula: (Covered + 0.5 * Partial) / Total * 100 Misinterpreted items count as 0 (same as Missing).

4b. Missing & Misinterpreted Items (sorted by impact)

❌ [Requirement summary]
   Source: "[exact quote from source]"
   Impact: High / Medium / Low
   Why it matters: [one sentence]

4c. Partial Coverage Items

🟡 [Requirement summary]
   Covered: [what the plan addresses]
   Gap:     [what's left out]

4d. Top Gaps (prioritized shortlist)

List the 5-10 highest-impact gaps most likely to produce a noticeably incomplete build.

Phase 5: Replan

After presenting the report:

1. If coverage >= 95%: Report the score. Note any remaining gaps as "acknowledged ambiguities." Ask if the user wants them addressed.
2. If coverage < 95%: Ask "Should I update the plan to address these gaps?"
3. When replanning:
   • Integrate missing items into the existing plan structure (not as an appendix)
   • Preserve everything already covered
   • Add specific implementation notes for previously missing items
   • Flag genuinely ambiguous items that need user clarification

Phase 6: Re-evaluate

After patching the plan, run Phases 2-4 again on the updated plan.

Stop when: coverage >= 95% OR remaining gaps are items the user acknowledges as intentionally unspecified.

Known gotchas

• The plan's framing is not the source of truth. Plans often restructure requirements in a way that sounds complete but drops items. Always check against the source.
• Implicit requirements get missed. "Like Spotlight" implies a global hotkey, float-above, escape-to-dismiss, fuzzy search — extract them.
• "Why" context is a requirement too. If the plan drops the rationale, downstream decisions get made without it.

Anti-patterns

• Treating one pass as enough. First-pass plans drop 20–40% of explicit requirements; re-evaluation after replanning catches the long tail.
• Merging related items for convenience. "User management" is not one requirement — break it into registration, login, password reset, roles, etc.
• Scoring on the plan's vocabulary instead of the source's. If the source says "feels like Spotlight" and the plan says "command palette UI", do not declare that covered without checking the specific behaviors implied by "Spotlight".

Validated patterns

• The score formula (Covered + 0.5×Partial) / Total × 100 calibrates well across plan sizes. 95% is the practical ceiling — below 90% almost always indicates a real gap, not a counting artifact.
• Sorting missing items by impact, not by source order, is the single biggest readability win in the report.

Self-improvement

This skill ships with a lightweight feedback loop. Adopt or ignore — the skill works without it.

Trigger a review when:

• The user explicitly disagrees with a coverage call (strongest signal — log immediately; 2–3 corrections on the same theme → promote to body).
• A golden case that previously passed starts failing.
• A novel input category recurs across sessions.
• learnings.md crosses ~100 bullets (consolidation time).
• The skill mis-triggers or fails to trigger.
• The Claude model version changes.

Consolidation pass (5–10 min, weekly or threshold-driven):

• Each entry gets one fate: apply, capture, or dismiss.
• Apply = merge into Known gotchas / Anti-patterns / Validated patterns.
• Capture = leave in learnings.md for now.
• Dismiss = delete.
• Bump last-consolidated: in frontmatter.