# Eval Design Forensics > Audit whether a paper's EVALUATION DESIGN actually measures what it claims and whether its reporting is complete β€” the validity layer family D (experiment-forensics) cannot reach. Three patterns: train/test leakage means the reported score may not measure generalization (HP-EVAL-LEAKAGE β€” adopts the Kapoor & Narayanan 8-type / 3-category leakage taxonomy; the illegitimate-proxy / sampling-bias / pretraining-contamination subtypes hand off as needs_external_check, naming but NEVER running Oren-2023 exchangeability / Shi-2023 Min-K% / Golchin-2023 Time-Travel / BIG-bench canary); a load-bearing LLM judge is conflicted (same model/family as a compared system) or unvalidated (no human-agreement, no bias control) (HP-JUDGE-VALIDITY); a declared condition/metric is dropped or switched to favor the method, or 'best' is chosen with no held-out set (HP-SELECTIVE-REPORTING). Verdict-bearing at L0/L1 from the DESCRIBED protocol β€” NOT repo-gated like experiment-forensics; L2 only CONFIRMS against split/preprocessing/resu - Skill: `wanshuiyin/eval-design-forensics` (Agent Skill) - Install (CLI): `npx skillmds@latest add wanshuiyin/eval-design-forensics` - Raw SKILL.md: https://api.skillmd.com/api/skills/wanshuiyin/eval-design-forensics/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: AI & ML - Author: wanshuiyin (https://skillmd.com/u/wanshuiyin) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/wanshuiyin/eval-design-forensics --- # Eval-Design Forensics β€” does the evaluation measure what the paper claims? Audit evaluation-design and reporting validity for: **$ARGUMENTS** (requires `claims.json` from `/evidence-ledger`). Emit span-anchored `eval-design-forensics.findings.json`. This skill computes **no verdict**. > πŸ”’ **Do not wrap this skill in `/loop`, `/schedule`, or `CronCreate`.** It is > verdict-bearing input β€” it proposes the findings the deterministic adjudicator > turns into the report. Re-firing it on a wall-clock timer adds no signal: its > output changes only when the **paper / ledger** changes (or a repo arrives, > raising the observability level), not with the clock. Schedule the *external wait > that precedes it* β€” ledger built (or artifacts released β†’ L2) β†’ audit **once**. > (Mirrors ARIS's external-cadence doctrine.) > Adapted from the ML-evaluation-methodology literature β€” the leakage taxonomy of > Kapoor & Narayanan (2023), the LLM-as-judge validity work (MT-Bench > self-enhancement, self-preference, position bias), and the "Show Your Work" / > reproducibility-checklist reporting norms β€” reframed to audit a **third party's** > evaluation. A favourite autoresearch shortcut is to report a number that is > arithmetically self-consistent (family A), runs real code against a real ground > truth (family D), and **still does not measure what it claims**: the protocol > leaks, the load-bearing metric is a conflicted/unvalidated LLM judge, or the > reporting quietly drops a declared condition. This skill is the constraint that > asks "is this a *valid measurement of the claim*?", pointed at a submission, and > it stays honest β€” leakage and under-reporting are usually **honest methodological > errors**, so every finding is a discrepancy to *clarify*, never an accusation. ## Why this exists An optimizing pipeline (or rushed human) treats the evaluation as a number to make go up, not a measurement to keep valid. The repeatable failure modes β€” distinct from "is the number real?" (family D) β€” are: - **Leakage** β€” the train/test boundary is broken (preprocessing fit before the split, no held-out set, duplicates across splits, a random split over time-ordered data, the same subject in both splits, an evaluated LLM that saw the benchmark in pretraining), so the reported score may not measure **generalization** at all. `HP-EVAL-LEAKAGE` - **Judge validity** β€” the headline rests on an automatic **LLM judge** that is *conflicted* (the same model/family as a compared system, so its preference for that system is the "evidence") or *unvalidated* (no human-agreement correlation, no position/length bias control). `HP-JUDGE-VALIDITY` - **Selective reporting** β€” a dataset / baseline / metric / seed-count the setup **explicitly declares** is dropped from the results, the metric is **switched** across tables to keep the method ahead, or "we report the best run/prompt/ checkpoint" with **no held-out selection set** (selecting on the test set). `HP-SELECTIVE-REPORTING` None of these is inherently misconduct β€” they are what an agent does when nothing forces a *valid* evaluation. The **stated** version is decidable at **L0/L1** from the described protocol; the **verified** version (real split/preprocessing/result files) deepens at **L2**. What this skill will **not** do is *guess*: three leakage subtypes are undecidable even with the repo and are handed off as `needs_external_check`, not invented (see below). ## Core principle **Ledger-anchored, span-verified, reviewerβ‰ adjudicator, honest about what it cannot settle.** Four properties: 1. **Anchor to a PAPER claim.** Every above-`info` finding cites a ledger `claim_id` and quotes a **verbatim span of that claim's `text_span`** (`references/integrity-forensics-contract.md` rules 1–2). The leak/judge/reporting tell lives in the *protocol / setup-description* β€” usually `method` and `scope` claims, with `comparison` / `number` for the judge metric and `caption` / `table_cell` / `baseline` for reporting. The anchor is whichever paper claim the finding undermines; a split-file `file:line`, a config, or a leaderboard date is **forensic context for the description**, never the anchor. 2. **The executor assembles facts; the reviewer judges.** At L2 the executor gathers **mechanical** split/preprocessing/judge/result facts (grep/hash β€” listing what exists is a fact, not a judgment) and may record one **public-record date fact** (a benchmark's release vs a model's cutoff, for the contamination FP guard). It passes **paths + the ledger + those facts + the checklist** to the reviewer and never pre-declares "this leaks" (`references/reviewer-independence.md`). The model **proposes**; `tools/adjudicate_findings.py` **decides**. This skill computes **no verdict**. 3. **Undecidable leakage subtypes β†’ hand off, don't guess.** An *illegitimate-proxy feature*, *sampling bias in the test set*, and *pretraining/benchmark contamination* are domain / black-box judgments not settleable from the PDF **or the repo**. Emit `verdict_local: needs_external_check` + `requires_external_check: true` (contract rule 6); **name** the external methods a domain check would use β€” exchangeability (Oren 2023), Min-K% Prob (Shi 2023), Time-Travel (Golchin 2023), BIG-bench canary strings β€” and **never run them**. 4. **Verdict-bearing at L0/L1; observability still caps the L2-confirm.** Unlike `experiment-forensics` (no eval code at L0/L1 β‡’ info-only), a **stated-tell** here is decided from the described protocol and emits `observability_level_required: 0`. The **L2 confirmation** of the same leak/ omission is a **separate** finding with `observability_level_required: 2` that is marked as needing L2 on a PDF-only run (`references/observability-levels.md`). So a PDF-only run keeps the stated-tell as a flag and the verification as an info "confirm-at-L2" pointer β€” never the reverse. ## How this differs from the other auditors (route correctly) This skill is the **L0/L1-stated / L2-verified** sibling of `baseline-comparison-audit` and `proof-derivation-forensics` (both verdict-bearing **without a repo**) β€” *not* the L2-only `experiment-forensics`. | Auditor | Question it answers | Level | |---------|---------------------|------| | **`eval-design-forensics`** (this) | **Is the evaluation a VALID measurement of the claim, and is the reporting complete?** (train/test leakage, conflicted/unvalidated LLM judge, declared-but-unreported / metric-switch / best-without-held-out) | **L0/L1 stated Β· L2 verified** | | `experiment-forensics` | Are the reported numbers what the **code** computes? (fake/derived GT, self-norm, phantom, dead metric) | L2 | | `consistency-audit` | Does the paper contradict ITSELF / described method = evaluated method? (owns `HP-AGG-DRIFT`, `HP-APPENDIX-CONTRA`, text-only `HP-SCOPE-INFLATE`) | L0 | | `baseline-comparison-audit` | Are the right baselines present, fairly tuned, and is "SOTA" earned? (owns `HP-MISSING-BASELINE`, `HP-SIG-OVERLAP`) | L0 stated / L2 verified | | `citation-forensics` | Do the cited papers exist and support the claim? | L0 | | `presentation-signals` | Surface "AI-flavor" hints (auxiliary, surface-class) | L0 | | `adversarial-case-builder` | Strongest evidence-bound rejection memo (no verdict weight) | any | **Do NOT raise here** (hand off instead): - **An LLM generating the GROUND-TRUTH labels/targets** (not judging outputs) β†’ `experiment-forensics` `HP-FAKE-GT` (L2). The clean split: a judge whose **preference IS the reported metric** is `HP-JUDGE-VALIDITY` (here, L0/L1 stated); a model that **fabricates the reference** the metric is computed against is `HP-FAKE-GT` (there, needs the code, L2). When unsure which, prefer the L2 route and set `needs_external_check`. - **best-reported-as-mean** (the aggregation lies) β†’ `consistency-audit` `HP-AGG-DRIFT`; **thin overall scope** with no comparison β†’ `consistency-audit` `HP-SCOPE-INFLATE`; **appendix-vs-main disagreement on the same quantity** β†’ `consistency-audit` `HP-APPENDIX-CONTRA`. - **A never-mentioned expected SOTA baseline** (completeness) β†’ `baseline-comparison-audit` `HP-MISSING-BASELINE`; a "consistently/across-the-board" comparison resting on **one dataset** β†’ `baseline-comparison-audit`'s single-dataset `HP-SIG-OVERLAP`. - **Whether a reported number matches the code** (fake GT, self-norm, phantom) β†’ `experiment-forensics` (L2); **whether a cited paper exists / is used in context** β†’ `citation-forensics`; **surface / AI-flavor** β†’ `presentation-signals`. `HP-SELECTIVE-REPORTING` is **scoped to declared-but-unreported / cherry-picked- among-shown** β€” the gap between what the setup *promised* and what the tables *deliver*. It never re-emits the four patterns above. ## The Kapoor & Narayanan leakage taxonomy (adopted β€” paraphrased) `HP-EVAL-LEAKAGE` adopts the **eight leakage types in three categories** of Kapoor & Narayanan (2023), paraphrased. The reviewer maps each finding to one type and records it in the `description`. | K&N category (the leakage **TYPE**) | The tell (subtypes) | This repo's **observability** | Common false positive | |---|---|---|---| | **L1 β€” no clean train/test separation** | (a) no held-out test set at all; (b) preprocessing (scaling / imputation / resampling) **fit on all data before the split**; (c) feature selection fit before the split; (d) duplicate / near-duplicate records across splits | **L0** stated / **L2** verified | a transductive / semi-supervised design where overlap is **intended and declared**; preprocessing fit on **train only**, then applied to test (the *correct* pattern) | | **L2 β€” illegitimate (proxy) feature** | a feature that stands in for the target, or would be unavailable at prediction time | **needs_external_check** (domain judgment) | a "proxy-looking" feature that is **genuinely available** at prediction time | | **L3 β€” test set not from the distribution of interest** | (a) **temporal** leakage (random split over time-ordered data / training on the future); (b) **non-independence** (same subject / patient / group in both splits); (c) **sampling bias** in the test set | (a),(b) **L0** stated / **L2** verified; (c) **needs_external_check** | a correctly time-respecting split; a standard fixed benchmark split the field uses | | **(LLM-specific) pretraining / benchmark contamination** | the evaluated model may have seen the public benchmark during pretraining | **needs_external_check** (black-box) β€” name Oren 2023 (exchangeability), Shi 2023 (Min-K%), Golchin 2023 (Time-Travel), BIG-bench canary; **never run them** | a benchmark released **after** the model's training cutoff, or a corpus **documented to exclude** it | > ⚠️ **Two scales β€” do not conflate them.** K&N's **L1 / L2 / L3** are leakage-*type* > labels (severity-ordered *categories of leak*). This repo's **L0 / L1 / L2** are > *observability* levels (what you can *see*: PDF / +source / +repo). They are > orthogonal. A K&N-**L1** preprocessing leak that is *stated* in the protocol is > decidable at observability-**L0**. Every finding carries **both**: the K&N type in > `description`, the observability in `observability_level_required`. ## Constants & Reviewer Calling Convention ``` REVIEWER_MODEL = gpt-5.6-sol # different family from executor (Claude) REVIEWER_REASONING = xhigh # always; effort never lowers reviewer quality REVIEWER_SANDBOX = read-only # detect-only; never mutate the paper REVIEWER_CWD = # so it can read claims.json + the protocol/source directly THREAD_POLICY = fresh mcp__codex__codex per PASS (and per entry on fan-out); NEVER mcp__codex__codex-reply across passes/entries (the bias guard) TAXONOMY_VERSION = 0.5 # references/hack-pattern-taxonomy.md (family H) LEAKAGE_TAXONOMY = Kapoor & Narayanan 2023 # 8 types / 3 categories β€” adopted, paraphrased PATTERNS_OWNED / ALLOWED = HP-EVAL-LEAKAGE, HP-JUDGE-VALIDITY, HP-SELECTIVE-REPORTING # emit ONLY these DIMENSION = evaluation # SKILL_TO_DIMENSION["eval-design-forensics"] FINDINGS_FILE = eval-design-forensics.findings.json FINDING_ID_NAMESPACE = ED### # distinct from F###/NUM###/HL### (consistency), EF### (experiment), BC### (baseline), PD### (proof) VERDICT_BEARING_AT = L0/L1 (stated-tells) # NOT repo-gated; L2 only CONFIRMS TRACE_POLICY = forensic (never silently dropped) TRACE_DIR = .aris/traces/eval-design-forensics/_run/ ``` - **Executor (Claude)** builds none of the judgment: it locates the ledger, extracts the evaluation surface (protocol / judge / declared-condition claims), at L2 gathers **mechanical** split/preprocessing/judge/result facts (grep/hash) and at most one **public-record date fact** for the contamination guard, passes **paths + the ledger + those facts + the checklist** to the reviewer, validates the reviewer's spans, and writes the findings file. It never summarizes the paper, pre-judges "this leaks", or leaks an opinion into the prompt (`reviewer-independence.md`). Passing a public release date (with its source) is the same allowed division `citation-forensics` (canonical metadata) and `baseline-comparison-audit` (leaderboard dates) use β€” reference facts, not hunches. - **Reviewer (codex / gpt-5.6-sol)** reads `claims.json` and the source (and, at L2, the split/preprocessing/judge/result files) directly from its `cwd`, decides which evaluations leak / rest on a conflicted-or-unvalidated judge / under-report, applies the known false-positive cases, and self-reports `false_positive_risk`. It is the evidence-extractor, not the judge. - **Fresh thread per pass.** Leakage (Step 3) and judge-validity + selective-reporting (Step 4) are **separate fresh** `mcp__codex__codex` calls. On `β€” effort: max` or many evaluation tracks, fan each track **entry** out into its own fresh call β€” never `codex-reply` carrying one entry's conclusion into another (the bias guard). `codex-reply` is intentionally absent from `allowed-tools`. --- ## Step 0 β€” Preconditions: locate the ledger, read the run level The ledger is the **only** structure this skill reasons over. Resolve it and read the observability level **L** and `paper_id` it was built at (each Bash block is self-contained β€” shell state does not persist, so re-derive paths every block): ```bash ROOT=$(git rev-parse --show-toplevel 2>/dev/null || pwd) # $ARGUMENTS is a paper-dir OR a claims.json path: LEDGER="$ARGUMENTS"; [ -d "$LEDGER" ] && LEDGER="$LEDGER/claims.json" # Only the NO-ARGUMENT case defaults to the CWD ledger. An EXPLICIT argument that # resolves to a missing claims.json must NOT silently fall back to $(pwd) β€” that # could audit the wrong paper; let the NO_LEDGER check below fire instead. [ -z "$ARGUMENTS" ] && LEDGER="$(pwd)/claims.json" python3 - "$LEDGER" <<'PY' import json, sys, os, collections p = sys.argv[1] if not os.path.isfile(p): sys.exit("NO_LEDGER: claims.json not found. Run /evidence-ledger FIRST " "(it writes artifact_manifest.json + claims.json).") d = json.load(open(p, encoding="utf-8")) claims = d.get("claims", []) by = collections.Counter(c.get("type") for c in claims) print("LEDGER =", os.path.abspath(p)) print("PAPER_DIR =", os.path.dirname(os.path.abspath(p)) or ".") print("PAPER_ID =", d.get("paper_id", "?")) print("RUN_LEVEL_L =", d.get("observability_level", 0)) print("CLAIMS =", len(claims), dict(by)) # applicability signal β€” method/scope carry the protocol; comparison/number the judge # metric; caption/table_cell/baseline the reported conditions: rel = sum(by.get(t, 0) for t in ("method", "scope", "comparison", "number", "table_cell", "caption", "baseline")) print("APPLICABLE =", "yes" if rel else "low (no protocol/scope/comparison/table claims)") PY ``` **Failure handling.** If `NO_LEDGER` is printed, stop and tell the user to run `/evidence-ledger` first β€” this skill never re-reads the raw PDF and invents its own structure (contract rule 1). Carry `L`, `PAPER_ID`, and the absolute `LEDGER` / `PAPER_DIR` into every step below. ## Step 1 β€” Extract the evaluation surface from the ledger (decide whether to run) Pull the claims this audit reasons over β€” the **protocol** (leakage anchors), the **judge** (validity anchors), and the **declared conditions** (reporting anchors) β€” and decide if there is anything to audit. This is a mechanical surface scan; the reviewer decides validity: ```bash LEDGER="" python3 - "$LEDGER" <<'PY' import json, re, sys, collections d = json.load(open(sys.argv[1], encoding="utf-8")) claims = d.get("claims", []) LEAK = re.compile(r"\b(train(?:ing)?[\s/_-]*(?:and[\s/_-]*)?test|train[\s/_-]*test|split|held?[\s-]*out|" r"cross[\s-]*validat|k-?fold|preprocess|standardi[sz]|normali[sz]|imput|" r"resampl|oversampl|smote|feature[\s-]*select|leak|duplicat|de-?dup|" r"temporal|time[\s-]*(?:series|order)|contaminat|pre-?train|data\s+split)\b", re.I) JUDGE = re.compile(r"\b(LLM[-\s]*as[-\s]*a?[-\s]*judge|as\s+(?:a\s+)?judge|automatic(?:ally)?\s+(?:judg|evaluat|scor|rat)|" r"GPT-?4o?|GPT-?3\.5|Claude|Gemini|win[\s-]*rate|pairwise|preference|" r"rated\s+by|scored\s+by|judged\s+by|LLM\s+(?:judge|evaluator|grader))\b", re.I) DECLARE = re.compile(r"\b(we\s+(?:evaluate|report|test|measure|use)|datasets?|benchmarks?|metrics?|" r"seeds?|over\s+\d+\s+(?:seed|run)|best\s+(?:checkpoint|prompt|run|model|epoch)|" r"five|four|three|\{[^}]*\})\b", re.I) leak_a, judge_a, report_a = [], [], [] for c in claims: t, span = c.get("type"), c.get("text_span", "") sec = c.get("location", {}).get("section", "?") if t in ("method", "scope") and LEAK.search(span): leak_a.append((c["claim_id"], t, sec, span[:160])) if t in ("comparison", "scope", "method", "number") and JUDGE.search(span): judge_a.append((c["claim_id"], t, sec, span[:160])) if t in ("scope", "method", "caption", "table_cell", "baseline") and DECLARE.search(span): report_a.append((c["claim_id"], t, sec, span[:160])) print(f"LEAKAGE anchors: {len(leak_a)} JUDGE anchors: {len(judge_a)} REPORTING anchors: {len(report_a)}") for tag, rows in (("leak", leak_a), ("judge", judge_a), ("report", report_a)): for cid, t, sec, sp in rows[:30]: print(f" [{tag}:{t}] {cid} [{sec}] {sp!r}") print("APPLICABLE =", "yes" if (leak_a or judge_a or report_a) else "no -> write [] and stop") PY ``` **Branch.** If **APPLICABLE = no** (no protocol / judge / declared-condition claims), this skill is **not applicable**: write an empty `eval-design-forensics.findings.json` (`[]`), record a one-line `NOT_APPLICABLE` reason in the trace (Step 7), and stop. **Silent skip is forbidden** β€” the orchestrator globs `*.findings.json` and expects the file to exist. Otherwise record the three anchor lists for the prompts. A purely mechanical grep of the source helps surface the protocol/table language (do **not** judge validity here β€” that is the reviewer's job): ```bash LEDGER="" grep -rInE 'train[ /_-]*test|split|held[ -]*out|cross[ -]*valid|preprocess|standardi|normali|impute|leak|duplicat|as a judge|win rate|pairwise|we (evaluate|report) on|best (checkpoint|prompt|run)' \ "$(dirname "$LEDGER")" --include='*.tex' --include='*.txt' 2>/dev/null | head -60 ``` ## Step 2 β€” Gather mechanical facts (L2 split/preprocessing/judge/result; optional date fact) Create the run's trace dir **now** β€” its first use is the facts file written just below, so it must exist before Step 7. Reuse this exact `RUNDIR` in Steps 3–7 (do **not** create a second one): ```bash DATE=$(date +%Y-%m-%d); N=1 while [ -d ".aris/traces/eval-design-forensics/${DATE}_run$(printf %02d $N)" ]; do N=$((N+1)); done RUNDIR=".aris/traces/eval-design-forensics/${DATE}_run$(printf %02d $N)"; mkdir -p "$RUNDIR" echo "RUNDIR = $RUNDIR" # carry this exact path forward (shell state does not persist) ``` **At L2 only** (repo + result files present), gather raw, uninterpreted facts β€” paths + grep/hash only, the same executor/reviewer division as `experiment-forensics`. Skip this block at L0/L1 (there is no code to read β€” the reviewer block gets "L<2: ..."): ```bash ROOT=$(git rev-parse --show-toplevel 2>/dev/null || pwd) PAPER_DIR=""; L=""; RUNDIR="" if [ "$L" = "2" ]; then # (a) split / preprocessing / dedup ordering FACTS (the K&N-L1 / L3 tells) -> leakage_grep.txt grep -rInE 'train_test_split|StratifiedKFold|KFold|GroupKFold|TimeSeriesSplit|\.split\(|train/val|holdout|hold-out|'\ 'StandardScaler|MinMaxScaler|fit_transform|\.fit\(|SimpleImputer|impute|SMOTE|resample|oversampl|'\ 'SelectKBest|feature_select|drop_duplicates|duplicated\(|dedup|shuffle=True|random_state' \ "$PAPER_DIR" --include='*.py' --include='*.ipynb' 2>/dev/null | head -80 > "$RUNDIR/leakage_grep.txt" # (b) LLM-judge calling code FACTS -> judge_grep.txt grep -rInE 'as_judge|llm_judge|judge_model|gpt-?4|gpt-?3\.5|claude|gemini|openai|anthropic|'\ 'pairwise|win_rate|preference|rate_response|score_response|annotate' \ "$PAPER_DIR" --include='*.py' --include='*.ipynb' --include='*.yaml' --include='*.json' 2>/dev/null | head -60 > "$RUNDIR/judge_grep.txt" # (c) which DECLARED conditions actually produced result files (the selective-reporting L2 confirm) -> reporting_grep.txt { find "$PAPER_DIR/results" "$PAPER_DIR/outputs" "$PAPER_DIR/logs" -type f \( -name '*.json' -o -name '*.csv' \) 2>/dev/null | sort | head -60 echo "## metric/dataset keys present in result files:" grep -rIhoE '"(dataset|benchmark|metric|seed|split|task)"[^,}]{0,40}' \ "$PAPER_DIR/results" "$PAPER_DIR/outputs" "$PAPER_DIR/logs" 2>/dev/null | sort -u | head -60; } > "$RUNDIR/reporting_grep.txt" # (d) reproducibility anchors: hash each discovered file (space-safe; tolerant of zero matches) { grep -rIlE 'split|scaler|judge|metric' "$PAPER_DIR" --include='*.py' 2>/dev/null | head -n 20 find "$PAPER_DIR" -maxdepth 3 -path '*results*' -name '*.json' 2>/dev/null | head -n 20; } \ | while IFS= read -r ff; do shasum -a 256 "$ff" 2>/dev/null; done > "$RUNDIR/hashes.txt" echo "L2 facts -> $RUNDIR/{leakage_grep,judge_grep,reporting_grep,hashes}.txt" else echo "L<2: stated-tell pass only (no split/judge/result files to read)." fi ``` **Optional contamination date fact (the FP guard, not a detector).** If a benchmark is named and an evaluated model's training cutoff is knowable, you MAY record **one** public-record date fact β€” `WebSearch`/`WebFetch` for " release date" and " training cutoff" β€” and write it (URL + access date) to `$RUNDIR/contamination_dates.json`. This is a **fact** that *suppresses* a false contamination flag (benchmark released after the cutoff β†’ legitimate); it is **never** a contamination *detector*. The skill never runs Min-K% / exchangeability / Time-Travel. **Failure handling.** No network β†’ skip the date fact; the reviewer treats contamination as `needs_external_check` regardless. Empty L2 greps (a thin repo) β†’ pass "L2 but no split/judge/result files found" to the reviewer so it does not invent a leak. ## Step 3 β€” Leakage pass (cross-model, fresh thread) β†’ HP-EVAL-LEAKAGE Open a **fresh** `mcp__codex__codex` thread (Reviewer Calling Convention). The reviewer reads `claims.json` from its `cwd` for the described protocol and, at L2, the split/ preprocessing files; every finding anchors to a ledger `claim_id`. Send EXACTLY (fill every `[ ... ]`): ``` mcp__codex__codex: model: gpt-5.6-sol config: {"model_reasoning_effort": "xhigh"} sandbox: read-only cwd: prompt: | You are a train/test-LEAKAGE forensics reviewer. You judge ONE thing: given the EVALUATION PROTOCOL this paper describes (and, at L2, the split/preprocessing code it ships), is there a leak that means the reported score may NOT measure generalization? You do NOT judge whether numbers are real (that needs the code and is another auditor) and you do NOT grade the paper. Describe a discrepancy to CHECK/CLARIFY, never an accusation β€” leakage is most often an HONEST methodological error. Hand off what you cannot ground. INPUTS (in your working directory β€” read them directly): - claims.json β€” the evidence ledger. The PROTOCOL/SPLIT/PREPROCESSING language lives in type:"method" and type:"scope" claims; the ONLY structure you reason over. Each claim = {claim_id, type, text_span (VERBATIM), location, value?}. You MAY re-open a source file (and, at L2, the split/preprocessing code) to confirm a span is real, but you may NOT introduce a claim not in the ledger. LEAKAGE ANCHOR TARGETS (protocol/scope claims β€” claim_id + verbatim text_span): [paste the LEAKAGE anchors from Step 1] L2 SPLIT/PREPROCESSING FACTS (raw grep/hash β€” uninterpreted; empty if L<2): [inline RUNDIR/leakage_grep.txt + RUNDIR/hashes.txt, or "L<2: no repo/code available"] CONTAMINATION DATE FACT (public record, if gathered β€” GIVEN data, not a verdict): [inline RUNDIR/contamination_dates.json, or "none gathered"] RUN OBSERVABILITY LEVEL L = . THE LEAKAGE TAXONOMY you map onto (Kapoor & Narayanan 2023 β€” 8 types / 3 categories; paraphrase the type in your description). ⚠️ K&N's L1/L2/L3 below are leakage-TYPE labels β€” they are NOT this repo's observability L0/L1/L2 (what you can SEE). Set observability_level_required from what you can SEE, and name the K&N type in text: - K&N L1 (no clean separation): (a) no held-out test; (b) preprocessing fit on ALL data BEFORE the split; (c) feature selection before the split; (d) duplicate / near-duplicate rows across splits. -> observability 0 (stated) / 2 (verified). - K&N L2 (illegitimate/proxy feature): a feature that proxies the target or is unavailable at prediction time. -> needs_external_check (domain judgment). - K&N L3 (test not from the distribution of interest): (a) temporal leakage (random split over time-ordered data / training on the future); (b) non-independence (same subject/patient/group in both splits) -> observability 0 / 2; (c) sampling bias in the test set -> needs_external_check. - Pretraining/benchmark CONTAMINATION of an evaluated LLM (the model may have seen the public benchmark in pretraining) -> needs_external_check. You may NAME the external methods a human would use (exchangeability, Oren 2023; Min-K% Prob, Shi 2023; Time-Travel, Golchin 2023; BIG-bench canary) but you do NOT run them. HARD RULES (a finding that breaks any of these is worthless): 1. ANCHOR. Every finding above "info" MUST carry >=1 evidence {claim_id, span} where claim_id EXISTS in claims.json and span is a VERBATIM whitespace-normalized SUBSTRING of THAT claim's text_span (no paraphrase). The anchor is the protocol/ split/preprocessing claim the leak undermines; a code file:line goes in `description`, never as the anchor. ALWAYS anchor β€” even a needs_external_check finding β€” so it stays navigable. 2. DISCREPANCY, NOT ACCUSATION. Never "reject", "fabricated", "the authors cheated". 3. OBSERVABILITY. A leak visible in the DESCRIBED protocol => observability_level_ required = 0 (this is verdict-bearing from a PDF). A leak only CONFIRMABLE from the split/preprocessing files => a SEPARATE finding with observability_level_required = 2 (an L0/L1 run reports it as needing L2 β€” that is correct). NEVER put the stated tell at 2. 4. HAND OFF THE 3 UNDECIDABLE SUBTYPES. illegitimate-proxy feature, sampling bias, and pretraining/benchmark contamination are NOT decidable from the PDF or the repo: set verdict_local "needs_external_check", requires_external_check true, severity "info", false_positive_risk "high", and name what a human should check. 5. HONEST FP. A declared transductive/semi-supervised overlap, a standard fixed benchmark split, preprocessing fit on TRAIN ONLY then applied to test, a correctly time-respecting split, a benchmark released AFTER the model's cutoff β€” these LOOK like leaks but are legitimate. Say so; if the protocol is under-described, prefer needs_external_check over a flag. 6. pattern_id MUST be HP-EVAL-LEAKAGE. SEVERITY DECISION (HP-EVAL-LEAKAGE): - unambiguous stated leak (e.g. "standardize all features, then split") that plausibly invalidates the HEADLINE generalization claim -> "critical", FP "low", observability 0, requires_external_check false. - a leak affecting a NON-headline result, or a stated tell that needs the code to confirm -> "major" (FP "medium"); the L2 confirmation is a separate observability-2 finding. - proxy / sampling-bias / contamination -> "info", needs_external_check, FP "high". OUTPUT: a single JSON array and NOTHING ELSE (no prose, no code fence). Each element conforms to schemas/finding.schema.json: {"finding_id":"ED001","skill":"eval-design-forensics","pattern_id":"HP-EVAL-LEAKAGE", "title":"short, neutral","description":"which K&N type + the protocol span it undermines + (L2) the split/preprocessing file:line","severity":"critical|major| minor|info","observability_level_required":0, "evidence":[{"claim_id":"C0xx","span":"verbatim substring", "location":{"file":"...","section":"..."}}], "verdict_local":"fail|warn|clean|needs_external_check", "requires_external_check":true|false,"false_positive_risk":"low|medium|high", "alternative_explanation_checked": "", "recommended_reviewer_action":"what to CHECK or ASK β€” never 'reject'"} An empty array [] is a valid, honest result (the protocol shows no leak). ``` Persist the raw response to the trace dir (Step 7) **before** parsing. **Failure handling:** MCP stall β†’ re-invoke the **identical** prompt as a fresh `mcp__codex__codex` (never `codex-reply`). Prose instead of JSON β†’ the Step 5 validator extracts the outermost `[...]`; if none, re-ask once "Output ONLY the JSON array." If the L2 facts were empty, bias every code-confirm item toward `needs_external_check` β€” never invent a split file. ## Step 4 β€” Judge-validity + Selective-reporting pass (cross-model, fresh thread) A **separate, new** `mcp__codex__codex` thread. Send EXACTLY (fill every `[ ... ]`): ``` mcp__codex__codex: model: gpt-5.6-sol config: {"model_reasoning_effort": "xhigh"} sandbox: read-only cwd: prompt: | You are an EVALUATION-VALIDITY and REPORTING-COMPLETENESS forensics reviewer. You judge two things: (1) does a headline metric rest on an LLM JUDGE that is conflicted or unvalidated? (2) does the reporting DROP a condition the setup declared, SWITCH a metric to favor the method, or select "best" with no held-out set? PROPOSE findings only β€” do NOT grade the paper; describe a discrepancy to CHECK/CLARIFY, never an accusation. You do NOT judge whether the numbers are real (another auditor). INPUTS (read directly in your working directory): - claims.json β€” the JUDGE protocol lives in type:"method"/"comparison"/"scope"/ "number"; the DECLARED conditions in type:"scope"/"method"; the reported tables in type:"table_cell"/"caption"/"number"/"baseline". The ONLY structure you navigate by; each claim = {claim_id, type, text_span (VERBATIM), location, value?}. You MAY re-open a source file (and, at L2, the judge/result files) to confirm a span. JUDGE ANCHOR TARGETS (claim_id + verbatim text_span): [paste the JUDGE anchors from Step 1] REPORTING ANCHOR TARGETS (declared-condition / table claims β€” claim_id + verbatim): [paste the REPORTING anchors from Step 1] L2 JUDGE + RESULT-FILE FACTS (raw grep β€” uninterpreted; empty if L<2): [inline RUNDIR/judge_grep.txt + RUNDIR/reporting_grep.txt, or "L<2: no repo available"] RUN OBSERVABILITY LEVEL L = . HARD RULES: 1. ANCHOR every finding above "info" to a real claim_id + a VERBATIM substring of that claim. For JUDGE-VALIDITY anchor to the judge-protocol claim naming the judge model (and the compared-systems claim showing the family overlap, as extra evidence). For SELECTIVE-REPORTING anchor to the setup-DECLARATION claim (what was promised). A config/result file:line is forensic detail for the description β€” NOT a valid anchor. 2. DISCREPANCY, NOT ACCUSATION. Say what to CHECK/ASK. Never "reject"/"faked". 3. OBSERVABILITY. Judge identity + the absence of reported validation are read off the described protocol => observability_level_required 0 (1 if only the source shows it). A declared-but-unreported condition is stated => 0; CONFIRMING the condition actually ran but went unreported needs the result files => a SEPARATE observability-2 finding. 4. HONEST FP. (judge) a judge validated against human agreement with bias controls reported; a judge from a clearly DIFFERENT family than every compared system AND not load-bearing (corroborated by human eval / standard metrics); a calibrated standard protocol. UNVALIDATED-ONLY is HIGH-FP β€” missing validation *reporting* is not proof none was done (may be in an appendix / a cited standard). (reporting) a declared condition omitted but explicitly justified ("full grid in the repo"); different tasks legitimately using different standard metrics; best-selection on a DECLARED held-out validation set; an honestly-labeled pilot. 5. pattern_id MUST be one of: HP-JUDGE-VALIDITY, HP-SELECTIVE-REPORTING. CHECKLIST (one finding per concrete discrepancy): 1. JUDGE VALIDITY [HP-JUDGE-VALIDITY] β€” a headline comparison rests on an automatic LLM judge, and either: (a) CONFLICTED β€” the judge is the same MODEL or model FAMILY as a compared system (especially the proposed one), so its preference for that system IS the evidence (self-enhancement / self-preference). This is the lower-FP STRUCTURAL case (family overlap is checkable) -> severity major, false_positive_risk "medium". observability 0. (b) UNVALIDATED β€” the LLM judge is load-bearing yet the paper reports NO human-agreement validation (no correlation / kappa vs humans) AND NO bias control (no position-swap, no length/verbosity control) -> severity major if the headline rests on it, minor otherwise; false_positive_risk "high" (caps at minor). observability 0. ROUTING: an LLM that generates the GROUND-TRUTH labels/targets (not judging outputs) is HP-FAKE-GT (experiment-forensics, L2) β€” do NOT raise it here; if unsure which, set verdict_local "needs_external_check". 2. SELECTIVE REPORTING [HP-SELECTIVE-REPORTING] β€” one of: (a) a dataset / baseline / metric / seed-count the SETUP EXPLICITLY DECLARES is then omitted from the results and is not in the appendix; (b) METRIC-SWITCHING across tables (Table 2 reports M where the method leads; Table 3 quietly switches to M' where it also leads) in a way that consistently favors the proposed method; (c) "we report the best checkpoint / prompt / run" with NO held-out selection set (selecting on the test set). severity major; critical if the omission/switch/selection is what PRODUCES the headline (false_positive_risk "low" when the declared-vs-reported gap is unambiguous); minor if peripheral. observability 0 (stated) / 2 (the result file shows the condition ran but went unreported β€” a SEPARATE finding). DE-DUP (do NOT emit these β€” route them): best-reported-as-mean -> HP-AGG-DRIFT (consistency-audit); thin overall scope with no comparison -> HP-SCOPE-INFLATE (consistency-audit); a never-mentioned expected baseline -> HP-MISSING-BASELINE (baseline-comparison-audit); appendix-vs-main on the SAME quantity -> HP-APPENDIX-CONTRA (consistency-audit). This pattern is ONLY declared-but-unreported / cherry-picked-among-shown. OUTPUT: a single JSON array and NOTHING ELSE (schemas/finding.schema.json), same shape as the leakage prompt; set finding_id "ED0xx", skill "eval-design-forensics". An empty array [] is valid and honest. Set requires_external_check only when you genuinely cannot settle a point at this level. ``` **Deepen at L2.** When `L == 2`, the judge/result facts let the reviewer promote a text-only suspicion to a confirmed finding (keep `observability_level_required: 0` if the text already showed the tell; use `2` only for what the files reveal). **Fan-out (optional, breadth).** On `β€” effort: max` or many evaluation tracks, issue the relevant checklist item **per track** as a separate fresh `mcp__codex__codex` call and concatenate the arrays β€” never `codex-reply`. Persist each raw reply to the trace (Step 7). **Failure handling:** identical to Step 3. ## Step 5 β€” Validate + anchor (the anti-hallucination gate) The executor enforces the **ANCHOR** gate (the one `tools/adjudicate_findings.py` re-applies, so an anchored finding you keep is not silently rejected downstream) plus eval-design-specific **owned-pattern + schema-hygiene + external-check** pre-filters, **before** keeping anything. The span must be a verbatim, whitespace-normalized **substring of** the cited claim (`span in base`, never `base in span` β€” appending hallucinated text to a real claim must fail). Pass **every** saved raw reviewer response (leakage + judge/reporting + any per-track fan-out files); they merge into one findings file with one `ED###` namespace: ```bash ROOT=$(git rev-parse --show-toplevel 2>/dev/null || pwd) LEDGER="" OUT="$(dirname "$LEDGER")/eval-design-forensics.findings.json" # args: LEDGER OUT then each saved raw reviewer response file from Steps 3–4: python3 - "$LEDGER" "$OUT" "" "" <<'PY' import json, re, sys, os ledger_path, out_path = sys.argv[1], sys.argv[2] resp_paths = [p for p in sys.argv[3:] if p and os.path.isfile(p)] def nw(s): # mirror adjudicator _norm_ws (whitespace only) return " ".join((s or "").split()) OWNED = {"HP-EVAL-LEAKAGE", "HP-JUDGE-VALIDITY", "HP-SELECTIVE-REPORTING"} # ALLOWED set ABOVE = {"critical", "major", "minor"} # OBS map β€” the canonical observability each owned pattern is decidable at (documentation; # the REVIEWER sets observability_level_required per finding, and the adjudicator does the # real req>run_level downgrade). stated-tell = 0; repo confirm = 2; the 3 leakage external # subtypes carry no level (needs_external_check -> info). OBS = {"HP-EVAL-LEAKAGE": "0 stated / 2 verified (proxy|sampling|contamination -> needs_external_check)", "HP-JUDGE-VALIDITY": "0/1 stated (2 may corroborate)", "HP-SELECTIVE-REPORTING": "0 stated / 2 verified"} ledger = json.load(open( …(truncated)