Eval Design Forensics

Eval-Design Forensics — does the evaluation measure what the paper claims?

Audit evaluation-design and reporting validity for: $ARGUMENTS (requires claims.json from /evidence-ledger). Emit span-anchored eval-design-forensics.findings.json. This skill computes no verdict.

🔒 Do not wrap this skill in /loop, /schedule, or CronCreate. It is verdict-bearing input — it proposes the findings the deterministic adjudicator turns into the report. Re-firing it on a wall-clock timer adds no signal: its output changes only when the paper / ledger changes (or a repo arrives, raising the observability level), not with the clock. Schedule the external wait that precedes it — ledger built (or artifacts released → L2) → audit once. (Mirrors ARIS's external-cadence doctrine.)

Adapted from the ML-evaluation-methodology literature — the leakage taxonomy of Kapoor & Narayanan (2023), the LLM-as-judge validity work (MT-Bench self-enhancement, self-preference, position bias), and the "Show Your Work" / reproducibility-checklist reporting norms — reframed to audit a third party's evaluation. A favourite autoresearch shortcut is to report a number that is arithmetically self-consistent (family A), runs real code against a real ground truth (family D), and still does not measure what it claims: the protocol leaks, the load-bearing metric is a conflicted/unvalidated LLM judge, or the reporting quietly drops a declared condition. This skill is the constraint that asks "is this a valid measurement of the claim?", pointed at a submission, and it stays honest — leakage and under-reporting are usually honest methodological errors, so every finding is a discrepancy to clarify, never an accusation.

Why this exists

An optimizing pipeline (or rushed human) treats the evaluation as a number to make go up, not a measurement to keep valid. The repeatable failure modes — distinct from "is the number real?" (family D) — are:

• Leakage — the train/test boundary is broken (preprocessing fit before the split, no held-out set, duplicates across splits, a random split over time-ordered data, the same subject in both splits, an evaluated LLM that saw the benchmark in pretraining), so the reported score may not measure generalization at all. HP-EVAL-LEAKAGE
• Judge validity — the headline rests on an automatic LLM judge that is conflicted (the same model/family as a compared system, so its preference for that system is the "evidence") or unvalidated (no human-agreement correlation, no position/length bias control). HP-JUDGE-VALIDITY
• Selective reporting — a dataset / baseline / metric / seed-count the setup explicitly declares is dropped from the results, the metric is switched across tables to keep the method ahead, or "we report the best run/prompt/ checkpoint" with no held-out selection set (selecting on the test set). HP-SELECTIVE-REPORTING

None of these is inherently misconduct — they are what an agent does when nothing forces a valid evaluation. The stated version is decidable at L0/L1 from the described protocol; the verified version (real split/preprocessing/result files) deepens at L2. What this skill will not do is guess: three leakage subtypes are undecidable even with the repo and are handed off as needs_external_check, not invented (see below).

Core principle

Ledger-anchored, span-verified, reviewer≠adjudicator, honest about what it cannot settle. Four properties:

1. Anchor to a PAPER claim. Every above-info finding cites a ledger claim_id and quotes a verbatim span of that claim's text_span (references/integrity-forensics-contract.md rules 1–2). The leak/judge/reporting tell lives in the protocol / setup-description — usually method and scope claims, with comparison / number for the judge metric and caption / table_cell / baseline for reporting. The anchor is whichever paper claim the finding undermines; a split-file file:line, a config, or a leaderboard date is forensic context for the description, never the anchor.
2. The executor assembles facts; the reviewer judges. At L2 the executor gathers mechanical split/preprocessing/judge/result facts (grep/hash — listing what exists is a fact, not a judgment) and may record one public-record date fact (a benchmark's release vs a model's cutoff, for the contamination FP guard). It passes paths + the ledger + those facts + the checklist to the reviewer and never pre-declares "this leaks" (references/reviewer-independence.md). The model proposes; tools/adjudicate_findings.py decides. This skill computes no verdict.
3. Undecidable leakage subtypes → hand off, don't guess. An illegitimate-proxy feature, sampling bias in the test set, and pretraining/benchmark contamination are domain / black-box judgments not settleable from the PDF or the repo. Emit verdict_local: needs_external_check + requires_external_check: true (contract rule 6); name the external methods a domain check would use — exchangeability (Oren 2023), Min-K% Prob (Shi 2023), Time-Travel (Golchin 2023), BIG-bench canary strings — and never run them.
4. Verdict-bearing at L0/L1; observability still caps the L2-confirm. Unlike experiment-forensics (no eval code at L0/L1 ⇒ info-only), a stated-tell here is decided from the described protocol and emits observability_level_required: 0. The L2 confirmation of the same leak/ omission is a separate finding with observability_level_required: 2 that is marked as needing L2 on a PDF-only run (references/observability-levels.md). So a PDF-only run keeps the stated-tell as a flag and the verification as an info "confirm-at-L2" pointer — never the reverse.

How this differs from the other auditors (route correctly)

This skill is the L0/L1-stated / L2-verified sibling of baseline-comparison-audit and proof-derivation-forensics (both verdict-bearing without a repo) — not the L2-only experiment-forensics.

Do NOT raise here (hand off instead):

• An LLM generating the GROUND-TRUTH labels/targets (not judging outputs) → experiment-forensics HP-FAKE-GT (L2). The clean split: a judge whose preference IS the reported metric is HP-JUDGE-VALIDITY (here, L0/L1 stated); a model that fabricates the reference the metric is computed against is HP-FAKE-GT (there, needs the code, L2). When unsure which, prefer the L2 route and set needs_external_check.
• best-reported-as-mean (the aggregation lies) → consistency-audit HP-AGG-DRIFT; thin overall scope with no comparison → consistency-audit HP-SCOPE-INFLATE; appendix-vs-main disagreement on the same quantity → consistency-audit HP-APPENDIX-CONTRA.
• A never-mentioned expected SOTA baseline (completeness) → baseline-comparison-audit HP-MISSING-BASELINE; a "consistently/across-the-board" comparison resting on one dataset → baseline-comparison-audit's single-dataset HP-SIG-OVERLAP.
• Whether a reported number matches the code (fake GT, self-norm, phantom) → experiment-forensics (L2); whether a cited paper exists / is used in context → citation-forensics; surface / AI-flavor → presentation-signals.

HP-SELECTIVE-REPORTING is scoped to declared-but-unreported / cherry-picked- among-shown — the gap between what the setup promised and what the tables deliver. It never re-emits the four patterns above.

The Kapoor & Narayanan leakage taxonomy (adopted — paraphrased)

HP-EVAL-LEAKAGE adopts the eight leakage types in three categories of Kapoor & Narayanan (2023), paraphrased. The reviewer maps each finding to one type and records it in the description.

⚠️ Two scales — do not conflate them. K&N's L1 / L2 / L3 are leakage-type labels (severity-ordered categories of leak). This repo's L0 / L1 / L2 are observability levels (what you can see: PDF / +source / +repo). They are orthogonal. A K&N-L1 preprocessing leak that is stated in the protocol is decidable at observability-L0. Every finding carries both: the K&N type in description, the observability in observability_level_required.

Constants & Reviewer Calling Convention

REVIEWER_MODEL        = gpt-5.6-sol                  # different family from executor (Claude)
REVIEWER_REASONING    = xhigh                    # always; effort never lowers reviewer quality
REVIEWER_SANDBOX      = read-only                # detect-only; never mutate the paper
REVIEWER_CWD          = <paper-dir>              # so it can read claims.json + the protocol/source directly
THREAD_POLICY         = fresh mcp__codex__codex per PASS (and per entry on fan-out);
                        NEVER mcp__codex__codex-reply across passes/entries (the bias guard)
TAXONOMY_VERSION      = 0.5                      # references/hack-pattern-taxonomy.md (family H)
LEAKAGE_TAXONOMY      = Kapoor & Narayanan 2023  # 8 types / 3 categories — adopted, paraphrased
PATTERNS_OWNED / ALLOWED = HP-EVAL-LEAKAGE, HP-JUDGE-VALIDITY, HP-SELECTIVE-REPORTING   # emit ONLY these
DIMENSION             = evaluation              # SKILL_TO_DIMENSION["eval-design-forensics"]
FINDINGS_FILE         = eval-design-forensics.findings.json
FINDING_ID_NAMESPACE  = ED###                    # distinct from F###/NUM###/HL### (consistency), EF### (experiment), BC### (baseline), PD### (proof)
VERDICT_BEARING_AT    = L0/L1 (stated-tells)     # NOT repo-gated; L2 only CONFIRMS
TRACE_POLICY          = forensic (never silently dropped)
TRACE_DIR             = .aris/traces/eval-design-forensics/<YYYY-MM-DD>_run<NN>/

• Executor (Claude) builds none of the judgment: it locates the ledger, extracts the evaluation surface (protocol / judge / declared-condition claims), at L2 gathers mechanical split/preprocessing/judge/result facts (grep/hash) and at most one public-record date fact for the contamination guard, passes **paths + the ledger
  • those facts + the checklist** to the reviewer, validates the reviewer's spans, and writes the findings file. It never summarizes the paper, pre-judges "this leaks", or leaks an opinion into the prompt (reviewer-independence.md). Passing a public release date (with its source) is the same allowed division citation-forensics (canonical metadata) and baseline-comparison-audit (leaderboard dates) use — reference facts, not hunches.
• Reviewer (codex / gpt-5.6-sol) reads claims.json and the source (and, at L2, the split/preprocessing/judge/result files) directly from its cwd, decides which evaluations leak / rest on a conflicted-or-unvalidated judge / under-report, applies the known false-positive cases, and self-reports false_positive_risk. It is the evidence-extractor, not the judge.
• Fresh thread per pass. Leakage (Step 3) and judge-validity + selective-reporting (Step 4) are separate fresh mcp__codex__codex calls. On — effort: max or many evaluation tracks, fan each track entry out into its own fresh call — never codex-reply carrying one entry's conclusion into another (the bias guard). codex-reply is intentionally absent from allowed-tools.

Step 0 — Preconditions: locate the ledger, read the run level

The ledger is the only structure this skill reasons over. Resolve it and read the observability level L and paper_id it was built at (each Bash block is self-contained — shell state does not persist, so re-derive paths every block):

ROOT=$(git rev-parse --show-toplevel 2>/dev/null || pwd)
# $ARGUMENTS is a paper-dir OR a claims.json path:
LEDGER="$ARGUMENTS"; [ -d "$LEDGER" ] && LEDGER="$LEDGER/claims.json"
# Only the NO-ARGUMENT case defaults to the CWD ledger. An EXPLICIT argument that
# resolves to a missing claims.json must NOT silently fall back to $(pwd) — that
# could audit the wrong paper; let the NO_LEDGER check below fire instead.
[ -z "$ARGUMENTS" ] && LEDGER="$(pwd)/claims.json"
python3 - "$LEDGER" <<'PY'
import json, sys, os, collections
p = sys.argv[1]
if not os.path.isfile(p):
    sys.exit("NO_LEDGER: claims.json not found. Run /evidence-ledger FIRST "
             "(it writes artifact_manifest.json + claims.json).")
d = json.load(open(p, encoding="utf-8"))
claims = d.get("claims", [])
by = collections.Counter(c.get("type") for c in claims)
print("LEDGER       =", os.path.abspath(p))
print("PAPER_DIR    =", os.path.dirname(os.path.abspath(p)) or ".")
print("PAPER_ID     =", d.get("paper_id", "?"))
print("RUN_LEVEL_L  =", d.get("observability_level", 0))
print("CLAIMS       =", len(claims), dict(by))
# applicability signal — method/scope carry the protocol; comparison/number the judge
# metric; caption/table_cell/baseline the reported conditions:
rel = sum(by.get(t, 0) for t in ("method", "scope", "comparison", "number", "table_cell", "caption", "baseline"))
print("APPLICABLE   =", "yes" if rel else "low (no protocol/scope/comparison/table claims)")
PY

Failure handling. If NO_LEDGER is printed, stop and tell the user to run /evidence-ledger first — this skill never re-reads the raw PDF and invents its own structure (contract rule 1). Carry L, PAPER_ID, and the absolute LEDGER / PAPER_DIR into every step below.

Step 1 — Extract the evaluation surface from the ledger (decide whether to run)

Pull the claims this audit reasons over — the protocol (leakage anchors), the judge (validity anchors), and the declared conditions (reporting anchors) — and decide if there is anything to audit. This is a mechanical surface scan; the reviewer decides validity:

LEDGER="<abs path to claims.json from Step 0>"
python3 - "$LEDGER" <<'PY'
import json, re, sys, collections
d = json.load(open(sys.argv[1], encoding="utf-8"))
claims = d.get("claims", [])
LEAK = re.compile(r"\b(train(?:ing)?[\s/_-]*(?:and[\s/_-]*)?test|train[\s/_-]*test|split|held?[\s-]*out|"
                  r"cross[\s-]*validat|k-?fold|preprocess|standardi[sz]|normali[sz]|imput|"
                  r"resampl|oversampl|smote|feature[\s-]*select|leak|duplicat|de-?dup|"
                  r"temporal|time[\s-]*(?:series|order)|contaminat|pre-?train|data\s+split)\b", re.I)
JUDGE = re.compile(r"\b(LLM[-\s]*as[-\s]*a?[-\s]*judge|as\s+(?:a\s+)?judge|automatic(?:ally)?\s+(?:judg|evaluat|scor|rat)|"
                   r"GPT-?4o?|GPT-?3\.5|Claude|Gemini|win[\s-]*rate|pairwise|preference|"
                   r"rated\s+by|scored\s+by|judged\s+by|LLM\s+(?:judge|evaluator|grader))\b", re.I)
DECLARE = re.compile(r"\b(we\s+(?:evaluate|report|test|measure|use)|datasets?|benchmarks?|metrics?|"
                     r"seeds?|over\s+\d+\s+(?:seed|run)|best\s+(?:checkpoint|prompt|run|model|epoch)|"
                     r"five|four|three|\{[^}]*\})\b", re.I)
leak_a, judge_a, report_a = [], [], []
for c in claims:
    t, span = c.get("type"), c.get("text_span", "")
    sec = c.get("location", {}).get("section", "?")
    if t in ("method", "scope") and LEAK.search(span):
        leak_a.append((c["claim_id"], t, sec, span[:160]))
    if t in ("comparison", "scope", "method", "number") and JUDGE.search(span):
        judge_a.append((c["claim_id"], t, sec, span[:160]))
    if t in ("scope", "method", "caption", "table_cell", "baseline") and DECLARE.search(span):
        report_a.append((c["claim_id"], t, sec, span[:160]))
print(f"LEAKAGE anchors: {len(leak_a)}  JUDGE anchors: {len(judge_a)}  REPORTING anchors: {len(report_a)}")
for tag, rows in (("leak", leak_a), ("judge", judge_a), ("report", report_a)):
    for cid, t, sec, sp in rows[:30]:
        print(f"  [{tag}:{t}] {cid} [{sec}] {sp!r}")
print("APPLICABLE   =", "yes" if (leak_a or judge_a or report_a) else "no -> write [] and stop")
PY

Branch. If APPLICABLE = no (no protocol / judge / declared-condition claims), this skill is not applicable: write an empty eval-design-forensics.findings.json ([]), record a one-line NOT_APPLICABLE reason in the trace (Step 7), and stop. Silent skip is forbidden — the orchestrator globs *.findings.json and expects the file to exist. Otherwise record the three anchor lists for the prompts. A purely mechanical grep of the source helps surface the protocol/table language (do not judge validity here — that is the reviewer's job):

LEDGER="<abs path to claims.json from Step 0>"
grep -rInE 'train[ /_-]*test|split|held[ -]*out|cross[ -]*valid|preprocess|standardi|normali|impute|leak|duplicat|as a judge|win rate|pairwise|we (evaluate|report) on|best (checkpoint|prompt|run)' \
    "$(dirname "$LEDGER")" --include='*.tex' --include='*.txt' 2>/dev/null | head -60

Step 2 — Gather mechanical facts (L2 split/preprocessing/judge/result; optional date fact)

Create the run's trace dir now — its first use is the facts file written just below, so it must exist before Step 7. Reuse this exact RUNDIR in Steps 3–7 (do not create a second one):

DATE=$(date +%Y-%m-%d); N=1
while [ -d ".aris/traces/eval-design-forensics/${DATE}_run$(printf %02d $N)" ]; do N=$((N+1)); done
RUNDIR=".aris/traces/eval-design-forensics/${DATE}_run$(printf %02d $N)"; mkdir -p "$RUNDIR"
echo "RUNDIR = $RUNDIR"   # carry this exact path forward (shell state does not persist)

At L2 only (repo + result files present), gather raw, uninterpreted facts — paths

• grep/hash only, the same executor/reviewer division as experiment-forensics. Skip this block at L0/L1 (there is no code to read — the reviewer block gets "L<2: ..."):

ROOT=$(git rev-parse --show-toplevel 2>/dev/null || pwd)
PAPER_DIR="<abs PAPER_DIR from Step 0>"; L="<L from Step 0>"; RUNDIR="<the RUNDIR above>"
if [ "$L" = "2" ]; then
  # (a) split / preprocessing / dedup ordering FACTS (the K&N-L1 / L3 tells) -> leakage_grep.txt
  grep -rInE 'train_test_split|StratifiedKFold|KFold|GroupKFold|TimeSeriesSplit|\.split\(|train/val|holdout|hold-out|'\
'StandardScaler|MinMaxScaler|fit_transform|\.fit\(|SimpleImputer|impute|SMOTE|resample|oversampl|'\
'SelectKBest|feature_select|drop_duplicates|duplicated\(|dedup|shuffle=True|random_state' \
      "$PAPER_DIR" --include='*.py' --include='*.ipynb' 2>/dev/null | head -80 > "$RUNDIR/leakage_grep.txt"
  # (b) LLM-judge calling code FACTS -> judge_grep.txt
  grep -rInE 'as_judge|llm_judge|judge_model|gpt-?4|gpt-?3\.5|claude|gemini|openai|anthropic|'\
'pairwise|win_rate|preference|rate_response|score_response|annotate' \
      "$PAPER_DIR" --include='*.py' --include='*.ipynb' --include='*.yaml' --include='*.json' 2>/dev/null | head -60 > "$RUNDIR/judge_grep.txt"
  # (c) which DECLARED conditions actually produced result files (the selective-reporting L2 confirm) -> reporting_grep.txt
  { find "$PAPER_DIR/results" "$PAPER_DIR/outputs" "$PAPER_DIR/logs" -type f \( -name '*.json' -o -name '*.csv' \) 2>/dev/null | sort | head -60
    echo "## metric/dataset keys present in result files:"
    grep -rIhoE '"(dataset|benchmark|metric|seed|split|task)"[^,}]{0,40}' \
      "$PAPER_DIR/results" "$PAPER_DIR/outputs" "$PAPER_DIR/logs" 2>/dev/null | sort -u | head -60; } > "$RUNDIR/reporting_grep.txt"
  # (d) reproducibility anchors: hash each discovered file (space-safe; tolerant of zero matches)
  { grep -rIlE 'split|scaler|judge|metric' "$PAPER_DIR" --include='*.py' 2>/dev/null | head -n 20
    find "$PAPER_DIR" -maxdepth 3 -path '*results*' -name '*.json' 2>/dev/null | head -n 20; } \
    | while IFS= read -r ff; do shasum -a 256 "$ff" 2>/dev/null; done > "$RUNDIR/hashes.txt"
  echo "L2 facts -> $RUNDIR/{leakage_grep,judge_grep,reporting_grep,hashes}.txt"
else
  echo "L<2: stated-tell pass only (no split/judge/result files to read)."
fi

Optional contamination date fact (the FP guard, not a detector). If a benchmark is named and an evaluated model's training cutoff is knowable, you MAY record one public-record date fact — WebSearch/WebFetch for " release date" and " training cutoff" — and write it (URL + access date) to $RUNDIR/contamination_dates.json. This is a fact that suppresses a false contamination flag (benchmark released after the cutoff → legitimate); it is never a contamination detector. The skill never runs Min-K% / exchangeability / Time-Travel.

Failure handling. No network → skip the date fact; the reviewer treats contamination as needs_external_check regardless. Empty L2 greps (a thin repo) → pass "L2 but no split/judge/result files found" to the reviewer so it does not invent a leak.

Step 3 — Leakage pass (cross-model, fresh thread) → HP-EVAL-LEAKAGE

Open a fresh mcp__codex__codex thread (Reviewer Calling Convention). The reviewer reads claims.json from its cwd for the described protocol and, at L2, the split/ preprocessing files; every finding anchors to a ledger claim_id. Send EXACTLY (fill every [ ... ]):

mcp__codex__codex:
  model: gpt-5.6-sol
  config: {"model_reasoning_effort": "xhigh"}
  sandbox: read-only
  cwd: <absolute PAPER_DIR from Step 0>
  prompt: |
    You are a train/test-LEAKAGE forensics reviewer. You judge ONE thing: given the
    EVALUATION PROTOCOL this paper describes (and, at L2, the split/preprocessing code
    it ships), is there a leak that means the reported score may NOT measure
    generalization? You do NOT judge whether numbers are real (that needs the code and
    is another auditor) and you do NOT grade the paper. Describe a discrepancy to
    CHECK/CLARIFY, never an accusation — leakage is most often an HONEST methodological
    error. Hand off what you cannot ground.

    INPUTS (in your working directory — read them directly):
      - claims.json — the evidence ledger. The PROTOCOL/SPLIT/PREPROCESSING language
        lives in type:"method" and type:"scope" claims; the ONLY structure you reason
        over. Each claim = {claim_id, type, text_span (VERBATIM), location, value?}. You
        MAY re-open a source file (and, at L2, the split/preprocessing code) to confirm a
        span is real, but you may NOT introduce a claim not in the ledger.
    LEAKAGE ANCHOR TARGETS (protocol/scope claims — claim_id + verbatim text_span):
      [paste the LEAKAGE anchors from Step 1]
    L2 SPLIT/PREPROCESSING FACTS (raw grep/hash — uninterpreted; empty if L<2):
      [inline RUNDIR/leakage_grep.txt + RUNDIR/hashes.txt, or "L<2: no repo/code available"]
    CONTAMINATION DATE FACT (public record, if gathered — GIVEN data, not a verdict):
      [inline RUNDIR/contamination_dates.json, or "none gathered"]
    RUN OBSERVABILITY LEVEL L = <L from Step 0>.

    THE LEAKAGE TAXONOMY you map onto (Kapoor & Narayanan 2023 — 8 types / 3 categories;
    paraphrase the type in your description). ⚠️ K&N's L1/L2/L3 below are leakage-TYPE
    labels — they are NOT this repo's observability L0/L1/L2 (what you can SEE). Set
    observability_level_required from what you can SEE, and name the K&N type in text:
      - K&N L1 (no clean separation): (a) no held-out test; (b) preprocessing fit on ALL
        data BEFORE the split; (c) feature selection before the split; (d) duplicate /
        near-duplicate rows across splits.  -> observability 0 (stated) / 2 (verified).
      - K&N L2 (illegitimate/proxy feature): a feature that proxies the target or is
        unavailable at prediction time.  -> needs_external_check (domain judgment).
      - K&N L3 (test not from the distribution of interest): (a) temporal leakage (random
        split over time-ordered data / training on the future); (b) non-independence
        (same subject/patient/group in both splits) -> observability 0 / 2; (c) sampling
        bias in the test set -> needs_external_check.
      - Pretraining/benchmark CONTAMINATION of an evaluated LLM (the model may have seen
        the public benchmark in pretraining) -> needs_external_check. You may NAME the
        external methods a human would use (exchangeability, Oren 2023; Min-K% Prob, Shi
        2023; Time-Travel, Golchin 2023; BIG-bench canary) but you do NOT run them.

    HARD RULES (a finding that breaks any of these is worthless):
    1. ANCHOR. Every finding above "info" MUST carry >=1 evidence {claim_id, span} where
       claim_id EXISTS in claims.json and span is a VERBATIM whitespace-normalized
       SUBSTRING of THAT claim's text_span (no paraphrase). The anchor is the protocol/
       split/preprocessing claim the leak undermines; a code file:line goes in
       `description`, never as the anchor. ALWAYS anchor — even a needs_external_check
       finding — so it stays navigable.
    2. DISCREPANCY, NOT ACCUSATION. Never "reject", "fabricated", "the authors cheated".
    3. OBSERVABILITY. A leak visible in the DESCRIBED protocol => observability_level_
       required = 0 (this is verdict-bearing from a PDF). A leak only CONFIRMABLE from the
       split/preprocessing files => a SEPARATE finding with observability_level_required = 2
       (an L0/L1 run reports it as needing L2 — that is correct). NEVER put the stated tell at 2.
    4. HAND OFF THE 3 UNDECIDABLE SUBTYPES. illegitimate-proxy feature, sampling bias, and
       pretraining/benchmark contamination are NOT decidable from the PDF or the repo:
       set verdict_local "needs_external_check", requires_external_check true, severity
       "info", false_positive_risk "high", and name what a human should check.
    5. HONEST FP. A declared transductive/semi-supervised overlap, a standard fixed
       benchmark split, preprocessing fit on TRAIN ONLY then applied to test, a correctly
       time-respecting split, a benchmark released AFTER the model's cutoff — these LOOK
       like leaks but are legitimate. Say so; if the protocol is under-described, prefer
       needs_external_check over a flag.
    6. pattern_id MUST be HP-EVAL-LEAKAGE.

    SEVERITY DECISION (HP-EVAL-LEAKAGE):
      - unambiguous stated leak (e.g. "standardize all features, then split") that
        plausibly invalidates the HEADLINE generalization claim -> "critical", FP "low",
        observability 0, requires_external_check false.
      - a leak affecting a NON-headline result, or a stated tell that needs the code to
        confirm -> "major" (FP "medium"); the L2 confirmation is a separate observability-2
        finding.
      - proxy / sampling-bias / contamination -> "info", needs_external_check, FP "high".

    OUTPUT: a single JSON array and NOTHING ELSE (no prose, no code fence). Each element
    conforms to schemas/finding.schema.json:
      {"finding_id":"ED001","skill":"eval-design-forensics","pattern_id":"HP-EVAL-LEAKAGE",
       "title":"short, neutral","description":"which K&N type + the protocol span it
       undermines + (L2) the split/preprocessing file:line","severity":"critical|major|
       minor|info","observability_level_required":0,
       "evidence":[{"claim_id":"C0xx","span":"verbatim substring",
                    "location":{"file":"...","section":"..."}}],
       "verdict_local":"fail|warn|clean|needs_external_check",
       "requires_external_check":true|false,"false_positive_risk":"low|medium|high",
       "alternative_explanation_checked": "<criticals ONLY — the benign readings you explicitly ruled out: rounding/display precision, unit or metric convention, statistical reporting convention, scope difference. Anchoring proves the text exists, NOT that your interpretation is right; a critical without this field demotes to major>",
       "recommended_reviewer_action":"what to CHECK or ASK — never 'reject'"}
    An empty array [] is a valid, honest result (the protocol shows no leak).

Persist the raw response to the trace dir (Step 7) before parsing. Failure handling: MCP stall → re-invoke the identical prompt as a fresh mcp__codex__codex (never codex-reply). Prose instead of JSON → the Step 5 validator extracts the outermost [...]; if none, re-ask once "Output ONLY the JSON array." If the L2 facts were empty, bias every code-confirm item toward needs_external_check — never invent a split file.

Step 4 — Judge-validity + Selective-reporting pass (cross-model, fresh thread)

A separate, new mcp__codex__codex thread. Send EXACTLY (fill every [ ... ]):

mcp__codex__codex:
  model: gpt-5.6-sol
  config: {"model_reasoning_effort": "xhigh"}
  sandbox: read-only
  cwd: <absolute PAPER_DIR from Step 0>
  prompt: |
    You are an EVALUATION-VALIDITY and REPORTING-COMPLETENESS forensics reviewer. You
    judge two things: (1) does a headline metric rest on an LLM JUDGE that is conflicted
    or unvalidated? (2) does the reporting DROP a condition the setup declared, SWITCH a
    metric to favor the method, or select "best" with no held-out set? PROPOSE findings
    only — do NOT grade the paper; describe a discrepancy to CHECK/CLARIFY, never an
    accusation. You do NOT judge whether the numbers are real (another auditor).

    INPUTS (read directly in your working directory):
      - claims.json — the JUDGE protocol lives in type:"method"/"comparison"/"scope"/
        "number"; the DECLARED conditions in type:"scope"/"method"; the reported tables in
        type:"table_cell"/"caption"/"number"/"baseline". The ONLY structure you navigate
        by; each claim = {claim_id, type, text_span (VERBATIM), location, value?}. You MAY
        re-open a source file (and, at L2, the judge/result files) to confirm a span.
    JUDGE ANCHOR TARGETS (claim_id + verbatim text_span):
      [paste the JUDGE anchors from Step 1]
    REPORTING ANCHOR TARGETS (declared-condition / table claims — claim_id + verbatim):
      [paste the REPORTING anchors from Step 1]
    L2 JUDGE + RESULT-FILE FACTS (raw grep — uninterpreted; empty if L<2):
      [inline RUNDIR/judge_grep.txt + RUNDIR/reporting_grep.txt, or "L<2: no repo available"]
    RUN OBSERVABILITY LEVEL L = <L from Step 0>.

    HARD RULES:
    1. ANCHOR every finding above "info" to a real claim_id + a VERBATIM substring of that
       claim. For JUDGE-VALIDITY anchor to the judge-protocol claim naming the judge model
       (and the compared-systems claim showing the family overlap, as extra evidence). For
       SELECTIVE-REPORTING anchor to the setup-DECLARATION claim (what was promised). A
       config/result file:line is forensic detail for the description — NOT a valid anchor.
    2. DISCREPANCY, NOT ACCUSATION. Say what to CHECK/ASK. Never "reject"/"faked".
    3. OBSERVABILITY. Judge identity + the absence of reported validation are read off the
       described protocol => observability_level_required 0 (1 if only the source shows it).
       A declared-but-unreported condition is stated => 0; CONFIRMING the condition actually
       ran but went unreported needs the result files => a SEPARATE observability-2 finding.
    4. HONEST FP. (judge) a judge validated against human agreement with bias controls
       reported; a judge from a clearly DIFFERENT family than every compared system AND not
       load-bearing (corroborated by human eval / standard metrics); a calibrated standard
       protocol. UNVALIDATED-ONLY is HIGH-FP — missing validation *reporting* is not proof
       none was done (may be in an appendix / a cited standard). (reporting) a declared
       condition omitted but explicitly justified ("full grid in the repo"); different
       tasks legitimately using different standard metrics; best-selection on a DECLARED
       held-out validation set; an honestly-labeled pilot.
    5. pattern_id MUST be one of: HP-JUDGE-VALIDITY, HP-SELECTIVE-REPORTING.

    CHECKLIST (one finding per concrete discrepancy):
     1. JUDGE VALIDITY [HP-JUDGE-VALIDITY] — a headline comparison rests on an automatic
        LLM judge, and either:
        (a) CONFLICTED — the judge is the same MODEL or model FAMILY as a compared system
            (especially the proposed one), so its preference for that system IS the
            evidence (self-enhancement / self-preference). This is the lower-FP STRUCTURAL
            case (family overlap is checkable) -> severity major, false_positive_risk
            "medium". observability 0.
        (b) UNVALIDATED — the LLM judge is load-bearing yet the paper reports NO
            human-agreement validation (no correlation / kappa vs humans) AND NO bias
            control (no position-swap, no length/verbosity control) -> severity major if
            the headline rests on it, minor otherwise; false_positive_risk "high" (caps at
            minor). observability 0.
        ROUTING: an LLM that generates the GROUND-TRUTH labels/targets (not judging
        outputs) is HP-FAKE-GT (experiment-forensics, L2) — do NOT raise it here; if
        unsure which, set verdict_local "needs_external_check".
     2. SELECTIVE REPORTING [HP-SELECTIVE-REPORTING] — one of:
        (a) a dataset / baseline / metric / seed-count the SETUP EXPLICITLY DECLARES is
            then omitted from the results and is not in the appendix;
        (b) METRIC-SWITCHING across tables (Table 2 reports M where the method leads;
            Table 3 quietly switches to M' where it also leads) in a way that consistently
            favors the proposed method;
        (c) "we report the best checkpoint / prompt / run" with NO held-out selection set
            (selecting on the test set).
        severity major; critical if the omission/switch/selection is what PRODUCES the
        headline (false_positive_risk "low" when the declared-vs-reported gap is
        unambiguous); minor if peripheral. observability 0 (stated) / 2 (the result file
        shows the condition ran but went unreported — a SEPARATE finding).
        DE-DUP (do NOT emit these — route them): best-reported-as-mean -> HP-AGG-DRIFT
        (consistency-audit); thin overall scope with no comparison -> HP-SCOPE-INFLATE
        (consistency-audit); a never-mentioned expected baseline -> HP-MISSING-BASELINE
        (baseline-comparison-audit); appendix-vs-main on the SAME quantity ->
        HP-APPENDIX-CONTRA (consistency-audit). This pattern is ONLY declared-but-unreported
        / cherry-picked-among-shown.

    OUTPUT: a single JSON array and NOTHING ELSE (schemas/finding.schema.json), same shape
    as the leakage prompt; set finding_id "ED0xx", skill "eval-design-forensics". An empty
    array [] is valid and honest. Set requires_external_check only when you genuinely
    cannot settle a point at this level.

Deepen at L2. When L == 2, the judge/result facts let the reviewer promote a text-only suspicion to a confirmed finding (keep observability_level_required: 0 if the text already showed the tell; use 2 only for what the files reveal). Fan-out (optional, breadth). On — effort: max or many evaluation tracks, issue the relevant checklist item per track as a separate fresh mcp__codex__codex call and concatenate the arrays — never codex-reply. Persist each raw reply to the trace (Step 7). Failure handling: identical to Step 3.

Step 5 — Validate + anchor (the anti-hallucination gate)

The executor enforces the ANCHOR gate (the one tools/adjudicate_findings.py re-applies, so an anchored finding you keep is not silently rejected downstream) plus eval-design-specific owned-pattern + schema-hygiene + external-check pre-filters, before keeping anything. The span must be a verbatim, whitespace-normalized substring of the cited claim (span in base, never base in span — appending hallucinated text to a real claim must fail). Pass every saved raw reviewer response (leakage + judge/reporting + any per-track fan-out files); they merge into one findings file with one ED### namespace:

ROOT=$(git rev-parse --show-toplevel 2>/dev/null || pwd)
LEDGER="<abs path to claims.json>"
OUT="$(dirname "$LEDGER")/eval-design-forensics.findings.json"
# args: LEDGER OUT then each saved raw reviewer response file from Steps 3–4:
python3 - "$LEDGER" "$OUT" "<resp_leakage.md>" "<resp_judge_reporting.md>" <<'PY'
import json, re, sys, os
ledger_path, out_path = sys.argv[1], sys.argv[2]
resp_paths = [p for p in sys.argv[3:] if p and os.path.isfile(p)]

def nw(s):                                   # mirror adjudicator _norm_ws (whitespace only)
    return " ".join((s or "").split())

OWNED = {"HP-EVAL-LEAKAGE", "HP-JUDGE-VALIDITY", "HP-SELECTIVE-REPORTING"}   # ALLOWED set
ABOVE = {"critical", "major", "minor"}
# OBS map — the canonical observability each owned pattern is decidable at (documentation;
# the REVIEWER sets observability_level_required per finding, and the adjudicator does the
# real req>run_level downgrade). stated-tell = 0; repo confirm = 2; the 3 leakage external
# subtypes carry no level (needs_external_check -> info).
OBS = {"HP-EVAL-LEAKAGE": "0 stated / 2 verified (proxy|sampling|contamination -> needs_external_check)",
       "HP-JUDGE-VALIDITY": "0/1 stated (2 may corroborate)",
       "HP-SELECTIVE-REPORTING": "0 stated / 2 verified"}

ledger = json.load(open(

…(truncated)