Validate Fidelity

Validate Fidelity — Spec & Visual Drift Detection

Compare specs and running implementations against the reverse-UI oracle to catch drift before it ships.

Input: $ARGUMENTS = --phase <spec|visual|all> [--domain <NNN-name>] [--focus <component>] [--oracle <path>]

Arguments

• --phase (required): spec | visual | all
  • spec — validate spec document against oracle JSON data
  • visual — validate running app against oracle screenshots + structure
  • all — run both phases sequentially for all matching domains
• --domain (optional): spec number or name, e.g., 004-home, 002-app-shell. If omitted, validates all domains that have both a spec and an oracle source.
• --focus (optional): component name to narrow scope, e.g., sidebar, module-card. Only validates requirements related to this component.
• --oracle (optional): path to reverse-UI output directory. Auto-detected if omitted.

Examples

/validate-fidelity --phase spec --domain 004-home
/validate-fidelity --phase visual --domain 004-home
/validate-fidelity --phase all
/validate-fidelity --phase spec --domain 002-app-shell --focus sidebar
/validate-fidelity --phase spec --oracle ./reverse-ui-localhost-3000

Phase 0: Parse Input & Locate Oracle

1. Parse $ARGUMENTS for --phase, --domain, --focus, --oracle.
2. If --phase is missing, ask the user which phase to run.
3. Locate the oracle in this priority order: a. --oracle flag (explicit path) b. specs/assets/manifest.json → read meta.baseUrl to derive reverse-ui-{domain}/ c. Glob for reverse-ui-*/manifest.json in current working directory d. Ask user if not found
4. Verify the oracle directory exists and contains manifest.json.
5. If --domain is specified, locate the corresponding spec file: specs/{domain}.md
6. If --domain is omitted, glob for all specs/*.md files and match each to an oracle page directory.
7. Check for component-kit availability: component-kit-{domain}/manifest.json — if present, use it for more precise validation.

Phase 1: Spec Validation (--phase spec)

Validate a spec document against the oracle's raw JSON data.

Step 1: Parse Spec Requirements

Read the spec .md file and extract each REQ-NNN-XX block. For each requirement, identify quantitative claims:

• Grid column counts
• Dimensions (width, height, gap)
• Color values (hex, rgb, rgba)
• Font families, sizes, weights
• Component counts (nav items, cards, sections)
• Text content (labels, titles, placeholders)
• Animation parameters (duration, easing, keyframes)
• Border radius, opacity, blur values
• Z-index values
• Breakpoint-specific behaviors

Step 2: Find Oracle Counterpart

For each claim, locate the matching data in the oracle:

If --focus is set, only process requirements whose REQ block or description mentions the focused component.

Step 3: Compare

For each claim, classify the result:

Tolerance rules:

• Dimensions: ±5% or ±2px (whichever is larger)
• Colors: exact hex/rgb match required (no "close enough" for colors)
• Counts: exact match required (3 columns ≠ 2 columns)
• Text: case-insensitive match, trimmed whitespace
• Font families: primary family must match (fallback stack can differ)

Step 4: Output Fidelity Report

Validating spec {domain} against oracle...

Source: reverse-ui-{domain}/pages/{page}/structure.json
        component-kit-{domain}/components/{component}.json (if available)

ERRORS (spec contradicts oracle):
  ❌ REQ-NNN-XX — {requirement title}
     Spec: {spec value}
     Oracle: {oracle value}
     Source: {json file} → {json path}

  ❌ REQ-NNN-XX — {requirement title}
     Spec: {spec value}
     Oracle: {oracle value or "not found"}
     Source: {json file} → {json path}

WARNINGS:
  ⚠️  {description of close match or unverifiable claim}
     Spec: {spec value}
     Oracle: {oracle value}

OMISSIONS (oracle has data, spec is silent):
  ℹ️  {description of missing spec coverage}
     Oracle: {oracle value}
     Source: {json file} → {json path}

PASSES:
  ✅ REQ-NNN-XX — {requirement title} ({brief match detail})
  ✅ REQ-NNN-XX — {requirement title} ({brief match detail})

Score: {pass}/{total} requirements match | {error} errors, {warn} warnings, {omission} omissions

Phase 2: Visual Validation (--phase visual)

Validate a running application against the oracle's screenshots and structure data.

Step 1: Verify Dev Server

Check if the dev server is running:

npm run dev -- --port 5173 &  # or detect from spec/oracle

If not running, ask the user to start it.

Step 2: Capture Current State

Use browser tools (mcp__claude-in-chrome__* or mcp__playwright__*) to:

1. Navigate to the page URL
2. Set viewport to 1440×900 (desktop)
3. Take a screenshot
4. Read the page structure (accessibility tree or DOM snapshot)
5. Extract computed styles for key elements (grid columns, dimensions, colors, fonts)

Step 3: Structural Comparison

Compare the captured DOM against the oracle's structure.json:

Step 4: Visual Comparison

Compare the captured screenshot against oracle screenshots:

Step 5: Output Visual Fidelity Report

Same format as the spec report, but with screenshot references:

Validating visual output for {domain}...

Captured: {screenshot path}
Oracle:   reverse-ui-{domain}/pages/{page}/screenshot-1440.png

STRUCTURAL ERRORS:
  ❌ Nav items — Expected {n} items, found {m}
     Oracle: structure.json → sections[0].children (sidebar nav)
  
  ❌ Module grid — Expected {n} columns, found {m}
     Oracle: structure.json → sections[2].styles.gridTemplateColumns

VISUAL ISSUES:
  ⚠️  {description of visual discrepancy}

PASSES:
  ✅ Sidebar present and dark-themed
  ✅ Hero heading text matches
  ✅ Search bar present with correct placeholder

Score: {pass}/{total} checks | {error} errors, {warn} warnings

Phase 3: All Validation (--phase all)

1. Glob for all specs: specs/*.md
2. For each spec, run Phase 1 (spec validation)
3. For each spec with a running page, run Phase 2 (visual validation)
4. Output combined report with per-domain scores and overall score

=== Fidelity Report — All Domains ===

002-app-shell:  Spec 8/10 ✅  |  Visual 7/9 ✅
004-home:       Spec 5/7  ⚠️  |  Visual 6/8 ⚠️
005-ideation:   Spec 9/9  ✅  |  Visual — (no dev server)

Overall: 22/26 spec (84%) | 13/17 visual (76%)
Top issues:
  1. ❌ 004-home REQ-004-06 — Module grid columns (2 vs 3)
  2. ❌ 004-home REQ-004-05 — Search bar gradient (invented)
  3. ❌ 002-app-shell REQ-002-03 — Nav item count (4 vs 8)

Oracle Path Resolution

The skill locates oracle data in this priority:

1. --oracle <path> — explicit, used as-is
2. specs/assets/manifest.json → read meta.baseUrl, derive domain, look for reverse-ui-{domain}/
3. reverse-ui-*/manifest.json — glob in cwd, use the first match (or ask if multiple)
4. component-kit-*/manifest.json — check alongside reverse-ui for enhanced validation
5. Ask user if nothing found

Limitations

• Spec validation is only as good as the requirement extraction — freeform prose without REQ-NNN-XX markers may be missed
• Visual validation requires a running dev server and browser automation tools
• Color comparison is exact (no perceptual color distance) — rgba(255,255,255,0.2) ≠ rgba(255,255,255,0.19)
• Screenshot comparison is structural, not pixel-diff — subtle spacing or shadow differences may not be caught
• Component-kit data (Phase 10 of reverse-ui) significantly improves validation precision — run /reverse-ui with --components first for best results
• WebGL/canvas content cannot be structurally compared in visual mode — only presence/absence is checked