# Trustworthy Cuj Scoring

> Use when building or running an agent eval, CUJ score, benchmark, or automated-improvement loop — and especially when a whole lane fails uniformly, a score looks too good or too bad to be true, you're about to trust a new grader/oracle/judge, or a scored harness invokes pnpm / npm / tsc / any .cmd subprocess on Windows. Symptoms: "the agent can't do X" reads as a capability gap, exit-127 / exit 127, a lane scores 0% or 100%, judge and oracle scores blended into one number, write-only-at-end eval crashed and lost work, ≥2 different failure modes on one harness, hill-climbing against a contaminated signal. Keywords: oracle, grader, LLM-as-judge, position bias, Cohen kappa, sealed holdout, decontamination, SNR, MDE, noise floor, checkpoint, status.jsonl, broken oracle, reward hacking, blind relaunch.

- Skill: `oimiragieo/trustworthy-cuj-scoring` (Agent Skill, multi-file: 2 files)
- Install (CLI): `npx skillmds@latest add oimiragieo/trustworthy-cuj-scoring`
- Raw SKILL.md: https://api.skillmd.com/api/skills/oimiragieo/trustworthy-cuj-scoring/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: oimiragieo (https://skillmd.com/u/oimiragieo)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/oimiragieo/trustworthy-cuj-scoring

---


# Trustworthy CUJ Scoring

## Overview

**A broken oracle reads as a capability gap.** Before you trust any CUJ/agent score, the
deterministic floor must be validated *bidirectionally* and the judge must be quarantined — a
grader that silently passes wrong answers (or fails right ones) makes the harness optimize
against a phantom, and any loop built on it hill-climbs in the wrong direction.

Build the score as a **cascade**: deterministic oracles are the gate; the LLM-judge is a
separate, last-resort layer for what no parser can express (semantic, taste, reward-hacking).

Full method, the LLM-judge statistics, and the 31 grounding sources live in **methodology.md** —
read it when building a harness, designing a judge protocol, or defending a reported number.

## When to Use

- Building or running an agent eval / CUJ score / agentic benchmark / automated-improvement loop.
- A whole lane fails (or passes) uniformly — check the **grader first**, not the model.
- A score looks too good or too bad to be true.
- You're about to trust a new oracle, grader, or judge.
- On Windows, the harness shells out to `pnpm` / `npm` / `tsc` / any `.cmd` wrapper.

**Not for:** rating a web surface (use `rate-fix-loop`); multi-benchmark eval-driver fault
isolation (a separate suite-resilience concern); the broader CUJ→cluster→fix loop (a separate
harness-evolution concern).

## The gates (checklist — run in order)

1. **Bidirectional oracle validation BEFORE any scored batch.** For every grader, run one
   **known-correct** input (MUST pass) AND one **known-wrong / empty** input (MUST fail). If
   either direction is wrong, the grader is broken — stop. (WebArena passed empty replies;
   tau-bench's do-nothing succeeded on 38% of tasks.)
1b. **Output-format / answer-equivalence validation.** Bidirectional validity proves the oracle
   DISCRIMINATES (right vs wrong); it does NOT prove it accepts every VALID answer. Also run a
   known-correct answer whose **format differs** from the reference (same numeric value at
   different precision, JSON keys reordered, `int` vs `str`, currency/percent symbols, thousands
   separators) — it MUST pass. If it fails, NORMALIZE before exact-match
   (strip/case/round/sort-keys/set-compare) OR pin the exact format in the task spec. Right-but-
   differently-formatted answers are false GAPs that DEFLATE the category — a 25–52 pp false-
   negative gap between strict and normalized scoring is measured in the wild (Ganapaneni 2026;
   CFMatch 2024). Receipts: a numeric-precision case + a JSON int-vs-str case (
   key-order) both scored GAP while trivially correct.
2. **Deterministic floor, THEN a quarantined judge.** Code graders (exit-code, regex, schema,
   AST, unit test, exact-match) are the gate. The judge runs only after the floor passes, only
   for what no parser can express, and its score is a **separate non-blended track** — never
   averaged into the deterministic headline. (Reward-hacking detection is the one thing only the
   judge can do: a test can't see a deleted assertion.)
3. **Per-task append-only checkpoint.** Write+flush one `status.jsonl` record per completed task;
   write `run_config.json` with the full task-ID list at run start; resume re-runs only
   errored+never-started. Never write-only-at-end.
4. **Pre-state the SNR / MDE / noise band** before the batch (N, baseline, minimum detectable
   effect, confidence). A result outside 2σ triggers a noise investigation, not a capability
   conclusion. Don't use the CLT under a few hundred datapoints.
5. **Sealed holdout + pre-registered gates.** Lock the pass/fail thresholds before any model is
   scored; test gold answers stay private. Treat a "clean" contamination scan as weak evidence —
   RL/GRPO training conceals contamination signals.
6. **≥2 distinct failure modes = rework, NOT relaunch.** Diagnose the root cause (subprocess,
   OS shell behavior, env leak) before re-running. Verify liveness before relaunch; ONE owner per
   run.
7. **Windows `.cmd` smoke-test.** `pnpm.cmd` / `tsc.cmd` via list-argv subprocess returns
   **exit-127** without `shutil.which()` or `shell=True` — it falsely fails the whole lane. Smoke-
   test the invocation on the target OS with a known-good pair before the batch.

## Quick Reference — run discipline

| Phase | Do |
|---|---|
| Pre-run | oracle self-test (good PASS + bad FAIL) · pre-register gates · state MDE/SNR · smoke-test `.cmd` on target OS · init `run_config.json` + `status.jsonl` |
| During | append+flush per-task checkpoint · `asyncio.wait_for(timeout=N)` per task · ONE owner |
| Post-run | judge as separate track (N≥3, AB+BA, Cohen's κ) · check vs pre-stated band · cluster failures · match fix-lever to cluster cardinality · retest before/after |
| Never | blind-relaunch after ≥2 failure modes · blend judge into the gate · trust a "clean" decon scan · single-trial judge on a high-stakes gate · CLT on small-N |

## Common Mistakes

- **Reading a uniform lane failure as a capability gap.** Check the grader first — the pnpm.cmd
  exit-127 trap made an entire UI lane (3/4) falsely fail and read as "the agent can't do UI."
- **Treating a deflated category score as a capability gap before ruling out oracle-strictness.**
  A trivially correct answer (easy mean, easy kv-get) that fails on numeric precision or JSON key
  order reads identically to a genuine capability failure. A 52 pp gap between strict and
  normalized scoring was measured in the wild (Ganapaneni 2026). Normalize the comparison first
  (strip/case/round/sort-keys/set-compare) and audit easy tasks that fail uniformly BEFORE drawing
  a capability conclusion. "Tool-use 40%" may be "40% minus N pp format deflation."
- **Blending judge and oracle into one headline number.** Judge variance pollutes the gate;
  quarantine it.
- **Single-trial judging on a gate.** Pairwise preferences flip 13.6% on average; 11 trials are
  needed for 95% fidelity. Use N≥3, randomized AB/BA, Cohen's κ — not raw agreement (which
  overstates discrimination 33–41 pp).
- **Write-only-at-end.** A crash or a killed subprocess loses every completed task. Checkpoint
  per task.
- **Optimizing before validating the signal.** A loop built on an unvalidated oracle gets *worse*
  at the real task while the metric appears to improve.