Cfn Ab Critic

CFN AB Critic

Purpose: Compare the build artifact ("ours") against an acceptance criterion's reference artifact BLINDLY — labels shuffled so the critic cannot tell which is ours — and emit a vote-manifest consumed by /cfn-vote-implement. The executable AC check still owns pass/fail; this is an additional quality layer that the AC opts into by carrying a reference key (validated by check-verifiable-static.sh check 1g).

Status is beta: the deterministic machinery (shuffle, manifest shape, routing) is mechanically tested; the LLM judgment step is driven by the agent following this doc and is not unit-testable.

The gap this fills

The failure class here is a builder grading its own homework: an implementation that meets the letter of the AC but is visibly worse than the reference artifact the AC pointed at. Stripping the "ours" label is what makes the comparison honest — the critic picks the better artifact without knowing which one it produced.

The blind mechanic (propose-never-write)

This skill runs as a Claude skill, so the LLM IS the critic. The blind is enforced mechanically by how execute.sh constructs the prompt and un-shuffles the verdict:

1. Phase 1 (no verdict yet). Run execute.sh. It computes a deterministic A/B label assignment per AC (see Determinism below), prints a prompt that shows ONLY artifact_A and artifact_B paths, and exits 0 without writing a manifest. The prompt reveals nothing about which side is ours.
2. You compare. Using the dispatch table below, ingest both artifacts. Pick the one that better satisfies the AC's quality bar. Produce a verdict per AC: {raw_winner: "A"|"B"|"tie", confidence: 0..1, biggest_gap: "one sentence"}.
3. Phase 2 (commit). Write the verdicts to a JSON file and re-run execute.sh --winner-file. It un-shuffles your raw A/B pick back to ours/reference/tie, emits the manifest, and exits per the routing table.

The propose-never-write rule: this skill NEVER modifies the build artifact. It only emits suggestions. Fixes route through /cfn-vote-implement, where the 3-agent vote decides whether they land.

Invocation

# Phase 1: emit the blinded prompt
/cfn-ab-critic --ac AC-7,AC-12 [--iteration N] [--verify VERIFY_<slug>.md]

# Phase 2: after you have written verdicts to /tmp/verdicts.json
/cfn-ab-critic --ac AC-7,AC-12 --winner-file /tmp/verdicts.json

Arguments:

Per-AC resolution precedence for artifacts: explicit --ours / --reference win, else --verify, else the AC is recorded status: blocked (the run never crashes on a missing artifact).

Artifact ingest dispatch

Symmetric, by extension. Apply the same dispatch to both artifact_A and artifact_B.

Video and remote non-HTML artifacts are blocked in v1 rather than compared badly. A blocked comparison is recorded, never crashes the run, and contributes to exit code 4 when every comparison is blocked.

Exit codes

blocked is a first-class outcome and is never reported as success. Per-AC blocks go into comparisons[] with a named blocked_reason; they never crash the whole run.

Manifest schema

Emitted at <project-root>/.cfn-cache/manifests/cfn-ab-critic-<nanosecond-timestamp>.json, in the shared cfn-vote-implement suggestion shape. The discovery glob in /cfn-vote-implement already lists cfn-ab-critic-*.json.

{
  "review_id": "ab-critic-<ns>",
  "source": "cfn-ab-critic",
  "generated_at": "<ISO-8601>",
  "status": "pending_review",
  "comparisons": [
    {"ac_id":"AC-7","ours_artifact":"<path>","reference_artifact":"<path>",
     "label_assignment":{"A":"ours","B":"reference"},
     "raw_winner":"A","winner":"ours","confidence":0.72,
     "biggest_gap":"<one sentence>","status":"compared"}
  ],
  "suggestions": [
    {"id":"S001","category":"reference-gap","tag":"polish",
     "one_liner":"<imperative>","title":"<...>","description":"<...>",
     "files":["AC-7"],"impact":"medium","effort":"medium",
     "suggested_approach":"<critic fix>","status":"pending"}
  ]
}

• comparisons[] is ALWAYS emitted (audit record, including blocked ones).
• label_assignment records which side was ours, so a reviewer can re-derive the shuffle and confirm the blind was not gamed.
• A suggestion is emitted ONLY when winner != "ours" OR confidence < threshold. When all pass: suggestions: [], exit 0.
• confidence is asserted to lie in [0,1]; out-of-range verdicts exit 3.

The suggestions[] field set is the folk-defined shared shape owned by cfn-dry-review/SKILL.md and consumed by cfn-vote-implement/SKILL.md (3/3, 2/3, 1/3, 0/3 routing). For the canonical field list (id, category, tag, one_liner, title, description, files, impact, effort, suggested_approach, status), reference those two files — this skill mirrors it exactly and does not redefine it.

cfn: suggestion schema copied (N+1)th time across producers; extract shared suggestion-schema.md when a 2nd drift bug lands

Tag routing

Determinism and audit trail

The A/B assignment is a pure SHA-256 derivative of (ac_id, iteration) — no $RANDOM, no Math.random, no wall-clock seeding:

HASH = sha256(ac_id "|" iteration)   # first 16 hex chars
ASSIGNMENT = (0xHASH[0:8]) % 2       # 0 -> A=ours,B=reference ; 1 -> A=reference,B=ours

Same (ac_id, iteration) ALWAYS reproduces the assignment. lib/shuffle.sh is sourceable so any reviewer can re-derive it:

source $HOME/.claude/skills/cfn-ab-critic/lib/shuffle.sh
label_assignment AC-7 0     # -> A=ours,B=reference

Because the assignment is reproducible, the value recorded in comparisons[].label_assignment is a real audit trail, not a claim. A reviewer re-runs the helper and confirms the manifest did not quietly swap labels to make "ours" win.

Rules

• Never let the builder grade itself. The phase-1 prompt reveals ONLY artifact_A / artifact_B.
• Never modify the build artifact. Propose, and let /cfn-vote-implement route the fix.
• Never report blocked as success.
• block-tagged findings are merge blockers regardless of vote outcome (same rule as cfn-persona-verify).
• Confidence outside [0,1] is a schema violation (exit 3), not a salvageable verdict.

v1 limitations

• Auto-wiring into the cfn-loop-task gate table lands in a follow-up; today invocation is manual via /cfn-ab-critic.
• --verify markdown parsing reads only the simple AC-<id>: reference=<path> ours=<path> line grammar. The JSON map form is the contract; richer verify docs should emit a JSON sidecar.
• Remote non-HTML artifacts and video are blocked, not compared.

Related

• check-verifiable-static.sh check 1g: the reference key contract this skill consumes.
• cfn-persona-verify/SKILL.md: structural sibling — same propose-never-write flow, same manifest-then-vote shape, same blocked is never pass rule.
• cfn-dry-review/SKILL.md: the canonical suggestion field set this skill mirrors.
• /cfn-vote-implement: consumes this manifest (discovery glob cfn-ab-critic-*.json is already listed).