Blind Critic

Blind Critic

Mechanism only. Dimensions and capture rules live in lenses/<lens>.md.

Core invariant: the critic subagent sees two anonymised artifacts and nothing else. No repo access, no diff, no build history, no knowledge of which side the AI produced. Give it any of those and it stops being a critic and becomes a self-review — which /deep REVIEW already has, and which is the thing this exists to improve on.

Gate

Skip silently, emitting VISUAL_SKIP:<reason>, when any of:

• DEEP_VISUAL_CRITIC_ENABLED=0
• No design/prototype/critic-map.json in the repo
• No pair in the map whose candidate.route is touched by this change

Skipping is the common case. Most /deep runs are not UI work and must pay nothing.

Round budget

Read .deep-{session8}/critic-state.json (create if absent):

{ "round": 1, "history": { "auth-login-desktop": [4.5] } }

• Rounds 1–2: blocking. Findings go to issues.json as severity: "blocking".
• Round 3+: advisory. Findings go to issues.json as severity: "advisory"; emit VISUAL_PASS.
• Plateau: if a pair's score fails to improve by >0.3 over its previous round, stop looping that pair and mark it advisory. Shumer's own run plateaued at ~5/10 — a loop without this is a token furnace.

Rounds are per-pair, not global. A pair that passes in round 1 is not re-run.

Steps

1. Capture

cd apps/web && CRITIC_ROUND=<round> pnpm critic:capture

Writes apps/web/e2e/critic/.rounds/round-<N>/<pair>-<viewport>/{A.png,B.png} and a key.json recording which letter is the candidate.

Never read key.json before dispatching subagents, and never quote it into a prompt. Read it only to interpret verdicts that have already come back.

2. Dispatch one blind subagent per pair

Run pairs concurrently — they are independent. Each subagent gets only:

• absolute paths to A.png and B.png
• the dimension table from lenses/<lens>.md
• the output contract below

It must not be told the route, the repo path, the pair id, which side is generated, or that one side is a design prototype.

Prompt skeleton:

Two interface screenshots, A and B. Read both. For each dimension below, judge which is stronger and why. Be a harsh critic — your default should be that both have real problems. Then state overall which single image is the better interface. You must pick one; "about the same" is only allowed when you genuinely cannot separate them. Return only the JSON contract. No prose outside it.

3. Output contract

{
  "winner": "A" | "B" | "tie",
  "score": { "A": 0.0, "B": 0.0 },
  "findings": [
    {
      "dimension": "hierarchy|spacing|states|density",
      "weaker": "A" | "B",
      "observation": "what differs, described perceptually",
      "severity": "blocking" | "advisory"
    }
  ]
}

observation describes what looks wrong, not where to fix it. The critic cannot see the codebase; a file path from it is a hallucination. FIX has the code and does the locating.

4. Resolve and record

Read key.json. Translate A/B back to candidate/reference.

On VISUAL_NEEDS_WORK: append findings to .deep-{session8}/issues.json, increment the pair's round, append its score to history, and let REVIEW route to FIX. The existing REVIEW→FIX→REVIEW loop does the iterating; this skill does not loop itself.

Always append the score trace to .deep-{session8}/test-results.json under visualCritic, including on pass. The trend across rounds is how you find out months from now whether this gate is earning its cost.

Calibration

Before trusting a verdict on real work, lenses/<lens>.md defines fixtures with known answers. Run them after any prompt change:

cd apps/web && pnpm critic:calibrate

An identical-pair fixture that returns anything other than a tie means the critic invents faults on demand, and every verdict it has produced is suspect.

Conventions

Mirrors the v11.2.0 merge-queue judge: verdict tokens grepped from output, an env opt-out, a bounded retry, and an auto-skip guard for inputs it cannot judge.