# Baseline Comparison Audit

> Audit whether a paper's baseline comparisons are COMPLETE, FAIR, and SIGNIFICANT: a required recent SOTA baseline is missing while 'best/SOTA' is claimed (HP-MISSING-BASELINE); a baseline is undertuned / given less compute-tuning-data, run at a mismatched config, or the equal-budget ablation-as-baseline is absent (HP-WEAK-BASELINE); 'outperforms' is asserted over overlapping error bars or with no variance/seeds (HP-SIG-OVERLAP); and a cross-row 'improves over baseline by X%' is arithmetically wrong (HP-DELTA-ERROR, cross-row form only). A versioned per-domain baseline profile + a live leaderboard/recency search are assembled by the EXECUTOR as structured facts; a fresh cross-model reviewer (gpt-5.5 xhigh, read-only, fresh thread per dimension) PROPOSES findings, each span-anchored to a ledger claim_id; tools/adjudicate_findings.py DECIDES the verdict. Works at L0 (stated comparisons) and deepens at L2 (configs/result files). A completeness question it cannot settle internally becomes needs_external_check, nev

- Skill: `majiayu000/baseline-comparison-audit` (Agent Skill, multi-file: 2 files)
- Install (CLI): `npx skillmds add majiayu000/baseline-comparison-audit`
- Raw SKILL.md: https://api.skillmd.com/api/skills/majiayu000/baseline-comparison-audit/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: majiayu000 (https://skillmd.com/u/majiayu000)
- Updated: 2026-09-09
- Page: https://skillmd.com/skills/majiayu000/baseline-comparison-audit

---


# Baseline Comparison Audit — is the comparison complete, fair, and significant?

Audit baseline-comparison integrity for: **$ARGUMENTS** (requires `claims.json`
from `/evidence-ledger`). Emit span-anchored `baseline-comparison-audit.findings.json`.
This skill computes **no verdict**.

> 🔒 **Do not wrap this skill in `/loop`, `/schedule`, or `CronCreate`.** It is
> verdict-bearing input — it proposes the findings the deterministic adjudicator
> turns into the report. Re-firing it on a wall-clock timer adds no signal: its
> output changes only when the **paper / ledger** (or the live leaderboard it
> cross-checks) changes, not with the clock. Schedule the *external wait that
> precedes it* — ledger built → audit **once**. (Mirrors ARIS's external-cadence
> doctrine.)

> Adapted from ARIS `paper-claim-audit` — its **scope-overclaim** and
> **delta-arithmetic** checks, reframed from "paper vs result files" to **"is the
> SOTA claim earned, and is the comparison a fair fight?"** — plus a per-domain
> baseline profile and a completeness / fairness / significance split. A favourite
> autoresearch shortcut is to claim SOTA while omitting the obvious recent baseline,
> to beat an undertuned one, or to write "outperforms" over error bars that overlap.
> This skill is the constraint that asks for the fair fight, pointed at a third
> party's submission, and it stays honest about what it cannot settle from a PDF.

## Why this exists

An autoresearch pipeline (or rushed human) optimises for the *headline* and treats
the comparison table as scaffolding to fill, not a fair experiment to run. The
repeatable failure modes:

- **Completeness** — "achieves state-of-the-art on GSM8K" while the obvious recent
  baseline a 2024–2026 reviewer expects is simply absent from the table, or the
  strong classical **floor** (BM25 for retrieval, GBDT for tabular, a linear/naive
  forecaster for time-series) is skipped while only weak neural baselines are beaten.
  `HP-MISSING-BASELINE`
- **Fairness** — the proposed method is tuned for 100 epochs / 5 seeds / extra data,
  the baseline is run at default settings for 10; or the compared rows use different
  backbones, splits, or eval protocols; or the single most informative baseline —
  the method's **own backbone with the new component removed, at an identical
  budget** — is missing. `HP-WEAK-BASELINE`
- **Significance** — "consistently outperforms" on a 0.3-point gap with overlapping
  error bars, with no variance / no seed count reported at all, or resting on a single
  dataset too thin for the "consistent / across-the-board" wording. `HP-SIG-OVERLAP`
- **Delta arithmetic** — "improves over the strongest baseline by 16%" when the
  baseline row is 73.1 and the proposed row is 78.0 (+6.7% relative / +4.9 points),
  the two operands sitting in *different* cells so the single-sentence deterministic
  pass cannot pair them. `HP-DELTA-ERROR` (cross-row form)

None of these is inherently misconduct — they are what an optimizing agent does when
nothing forces a fair comparison. The *stated* version is decidable at **L0** from
the manuscript; the *verified* version (real configs, real seeds) deepens at **L2**.
What this skill will **not** do is *guess*: where no domain profile exists and the
leaderboard search is inconclusive, the completeness question is handed off as
`needs_external_check`, not invented.

## Core principle

**Ledger-anchored, span-verified, reviewer≠adjudicator, honest about what it cannot
settle.** Four properties:

1. **Anchor to a PAPER claim.** Every above-`info` finding cites a ledger `claim_id`
   and quotes a **verbatim span of that claim's `text_span`**
   (`references/integrity-forensics-contract.md` rules 1–2). The "outperforms / SOTA
   / best / first" language lives in `comparison` and `scope` claims; the reported
   baseline set lives in `baseline` claims; values/table rows in `number` /
   `table_cell` claims. The anchor is whichever paper claim the finding undermines —
   the expected-baseline list, a leaderboard URL, or a config `file:line` are
   **forensic context for the description**, never the anchor.
2. **The executor assembles facts; the reviewer judges.** The profile + a live
   `WebSearch`/`WebFetch` give a *candidate expected-baseline set with sources and
   dates*; the executor passes it as **structured input** and never pre-declares
   "baseline X is missing" (`references/reviewer-independence.md`). The model
   **proposes**; `tools/adjudicate_findings.py` **decides**. This skill computes **no
   verdict**.
3. **Unsettleable completeness → hand off, don't guess.** "Is this really SOTA /
   first / the right baseline set?" cannot be closed from inside the paper. Unless an
   omission is *unambiguous, sourced, same-benchmark, and pre-dating*, emit
   `verdict_local: needs_external_check` + `requires_external_check: true`, not a flag
   (contract rule 6).
4. **Observability caps severity.** Stated-comparison checks are L0; a fairness
   finding that needs the actual config/seed files is `observability_level_required:
   2` and auto-demotes on a PDF-only run (`references/observability-levels.md`).

## How this differs from the other auditors (route correctly)

| Auditor | Question it answers | Level |
|---------|---------------------|------|
| `consistency-audit` | Does the paper contradict ITSELF / described method = evaluated method? (owns text-only `HP-SCOPE-INFLATE` + single-sentence `HP-DELTA-ERROR`) | L0 |
| `experiment-forensics` | Are the reported numbers what the code actually computes? (fake GT, self-norm, phantom) | L2 |
| **`baseline-comparison-audit`** (this) | **Are the right baselines present (completeness), fairly tuned/configured (fairness), and is "outperforms/SOTA" statistically earned (significance)?** | **L0 stated / L2 verified** |
| `citation-forensics` | Do the cited baseline papers exist and support the claim? | L0 |
| `presentation-signals` | Surface "AI-flavor" hints (auxiliary, capped at minor) | L0 |
| `adversarial-case-builder` | Strongest evidence-bound rejection memo (no verdict weight) | any |

**Do NOT raise here** (hand off instead): generic in-text scope inflation
("comprehensive / extensive / robust" decoupled from a SOTA/comparison claim) →
`consistency-audit` owns `HP-SCOPE-INFLATE`; a single-sentence "from A to B, X%"
delta whose operands and stated value sit in **one** sentence → already caught
deterministically by `consistency-audit` (do **not** re-emit — Step 5 dedups);
whether a baseline *number* matches the repo/code → `experiment-forensics` (L2);
whether a *cited* baseline paper exists / is used in-context → `citation-forensics`;
surface / AI-flavor → `presentation-signals`. This skill **never** emits an
F-pattern.

## Per-domain baseline profile (`PROFILE_VERSION = 0.1` — a SEED prior, always verified live)

The expected baseline set a competent 2024–2026 reviewer carries into the table. It
is **advisory** and deliberately at the level of *families / floors* (not pinned
method names that go stale); the **live `WebSearch`/`WebFetch` leaderboard check
(Step 2) is the authoritative cross-check** — the profile only seeds the question.
The **Fairness control** column names the matched-budget axis a `HP-WEAK-BASELINE`
finding turns on; the **Variance norm** column is what `HP-SIG-OVERLAP` turns on.

| Domain / benchmark | Expected baseline families (**bold = the easy-to-skip floor**) | Fairness control (matched-budget axis) | Variance norm (significance) |
|---|---|---|---|
| LLM reasoning / QA — GSM8K, MATH, MMLU, BBH, GPQA | a current frontier model (Llama-3.x, Qwen2.5, DeepSeek) + the prior method on the same benchmark; **strong CoT / self-consistency on the same base** | same base model; identical #shots, decoding (temp / SC samples), tool access, finetune data | variance over prompts/seeds for small gaps |
| Image classification — ImageNet-1k | a recent strong backbone at matched params/FLOPs (ConvNeXt-V2, DeiT-III, Swin-V2, MAE-ViT); **a well-tuned modern CNN** | params, FLOPs, input res, epochs, augmentation, pretrain data | single run common; ±std if pretraining differs |
| Detection / segmentation — COCO, ADE20K | a recent strong detector/segmenter at the same backbone & schedule (DINO, Co-DETR, ViTDet, Mask2Former); **a strong one-stage baseline** | backbone, schedule (1×/3×), input scale, extra data | single run common; ±std on mIoU if available |
| Machine translation — WMT | tuned Transformer-big + a recent NMT/LLM-MT system; **report COMET, not only BLEU** | data, model size, beam, vocab; same test split + (de)tok protocol | bootstrap CI on BLEU/COMET |
| Generation — FID on ImageNet/COCO, GenEval | a recent strong generator at matched sampling budget (DiT, EDM2, U-ViT, LDM); **report precision/recall, not only FID** | NFE/sampler, params, guidance; identical FID protocol (#samples, ref stats) | FID over a fixed sample size; seed/sample noise |
| Retrieval / RAG — BEIR, MTEB, NQ | a strong dense retriever + prior SOTA; **BM25 (the lexical floor)** | same corpus, index, eval protocol (full vs sampled negatives) | per-query bootstrap CI |
| Tabular learning | a strong recent DL-tab model + prior SOTA; **a well-tuned GBDT (XGBoost/LightGBM/CatBoost) — it MUST be tuned** | HPO-budget parity, features, splits | std over folds/seeds |
| Time-series forecasting | a strong recent forecaster + prior SOTA; **a linear / naive-seasonal baseline** | lookback, horizon, normalization, splits | std over windows/seeds |
| RL — control (MuJoCo/DMC), offline (D4RL), Atari | tuned SAC/TD3/PPO (online), CQL/IQL/Decision-Transformer (offline), Rainbow/IQN/DrQ/SPR (Atari); **a well-tuned standard algorithm** | env steps / frames / dataset, net size, #eval seeds & episodes | ≥5 seeds + std/IQM (rliable CI) |
| Code generation — HumanEval, MBPP, LiveCodeBench | a current frontier code LLM + prior SOTA + a same-size open base; **the base model w/o the proposed scaffold** | model size, #shots, decoding, contamination window | variance over samples (pass@k seeds) |
| Speech ASR — LibriSpeech | a Whisper-class / Conformer system + prior SOTA | training data, decoding / LM | WER ±CI if available |
| Graph — OGB | a strong GNN family + the prior OGB-leaderboard entry | features, splits | std over seeds |

**Cross-domain control (always applicable, even off-profile):** the single most
informative baseline is the proposed method's **own backbone / base model with the
new component removed, run at an identical budget** — the ablation-as-baseline. Its
absence, or an unequal budget for it, is the most common fairness failure and is
checkable for *any* paper, profile row or not.

**Honesty rule (load-bearing): no profile row + inconclusive search ⇒ NO guessed
"missing baseline".** Run the *fairness* + *significance* checks (which need no
profile) and emit the completeness question as `needs_external_check`. The profile
seeds a *question*, never a detector.

## Constants & Reviewer Calling Convention

```
REVIEWER_MODEL        = gpt-5.5                  # different family from executor (Claude)
REVIEWER_REASONING    = xhigh                    # always; effort never lowers reviewer quality
REVIEWER_SANDBOX      = read-only                # detect-only; never mutate the paper
REVIEWER_CWD          = <paper-dir>              # so it can read claims.json + sources directly
THREAD_POLICY         = fresh mcp__codex__codex per DIMENSION (and per entry on fan-out);
                        NEVER mcp__codex__codex-reply across dimensions/entries
TAXONOMY_VERSION      = 0.5                      # references/hack-pattern-taxonomy.md
PROFILE_VERSION       = 0.1                      # the per-domain baseline profile above (advisory)
PATTERNS_OWNED        = HP-MISSING-BASELINE, HP-WEAK-BASELINE, HP-SIG-OVERLAP,
                        HP-DELTA-ERROR (cross-row comparison form only — see Step 4),
                        HP-RESOURCE-IDENTITY-MISMATCH (named dataset/model/benchmark vs its
                        public record — HF card / Papers-with-Code; gather-facts-then-judge,
                        observability_level_required 0; FP-suppress subset/variant/version)
FINDINGS_FILE         = baseline-comparison-audit.findings.json
FINDING_ID_NAMESPACE  = BC###                    # distinct from F###/NUM###/HL### (consistency), EF### (experiment)
TRACE_POLICY          = forensic (never silently dropped)
TRACE_DIR             = .aris/traces/baseline-comparison-audit/<YYYY-MM-DD>_run<NN>/
```

- **Executor (Claude)** builds none of the judgment: it locates the ledger, extracts
  the comparison surface, assembles the candidate expected set **with sources and
  publication dates**, at L2 gathers **mechanical config/result facts** (grep/hash —
  listing what exists is a fact, not a judgment), passes **paths + the ledger + those
  facts + the checklist** to the reviewer, validates the reviewer's spans, and writes
  the findings file. It never summarizes the paper, pre-judges "X is missing", or
  leaks an opinion into the prompt (`reviewer-independence.md`). Passing what a public
  leaderboard says (with its date) is the same allowed division `experiment-forensics`
  (grep/hash facts) and `citation-forensics` (canonical metadata) use — reference
  facts, not hunches about the manuscript.
- **Reviewer (codex / gpt-5.5)** reads `claims.json` and the sources, decides which
  comparisons are incomplete / unfair / not significant, applies the known
  false-positive cases, and self-reports `false_positive_risk`. It is the
  evidence-extractor, not the judge.
- **Fresh thread per dimension.** Completeness (Step 3) and fairness + significance +
  delta (Step 4) are **separate fresh** `mcp__codex__codex` calls. On `— effort: max`
  or many comparison rows, fan each comparison **entry** out into its own fresh call —
  never `codex-reply` carrying one entry's conclusion into another (the bias guard).
  `codex-reply` is intentionally absent from `allowed-tools`.

---

## Step 0 — Preconditions: locate the ledger, read the run level

The ledger is the **only** structure this skill reasons over. Resolve it and read
the observability level **L** and `paper_id` it was built at (each Bash block is
self-contained — shell state does not persist between calls, so re-derive paths every
block):

```bash
ROOT=$(git rev-parse --show-toplevel 2>/dev/null || pwd)
# $ARGUMENTS is a paper-dir OR a claims.json path:
LEDGER="$ARGUMENTS"; [ -d "$LEDGER" ] && LEDGER="$LEDGER/claims.json"
# Only the NO-ARGUMENT case defaults to the CWD ledger. An EXPLICIT argument that
# resolves to a missing claims.json must NOT silently fall back to $(pwd) — that
# could audit the wrong paper; let the NO_LEDGER check below fire instead.
[ -z "$ARGUMENTS" ] && LEDGER="$(pwd)/claims.json"
python3 - "$LEDGER" <<'PY'
import json, sys, os, collections
p = sys.argv[1]
if not os.path.isfile(p):
    sys.exit("NO_LEDGER: claims.json not found. Run /evidence-ledger FIRST "
             "(it writes artifact_manifest.json + claims.json).")
d = json.load(open(p, encoding="utf-8"))
claims = d.get("claims", [])
by = collections.Counter(c.get("type") for c in claims)
print("LEDGER       =", os.path.abspath(p))
print("PAPER_DIR    =", os.path.dirname(os.path.abspath(p)) or ".")
print("PAPER_ID     =", d.get("paper_id", "?"))
print("RUN_LEVEL_L  =", d.get("observability_level", 0))
print("CLAIMS       =", len(claims), dict(by))
# applicability signal — comparison / scope / baseline / table_cell are this skill's inputs:
rel = sum(by.get(t, 0) for t in ("comparison", "scope", "baseline", "table_cell"))
print("APPLICABLE   =", "yes" if rel else "low (no comparison/scope/baseline/table claims)")
PY
```

**Failure handling.** If `NO_LEDGER` is printed, stop and tell the user to run
`/evidence-ledger` first — this skill never re-reads the raw PDF and invents its own
structure (contract rule 1). Carry `L`, `PAPER_ID`, and the absolute `LEDGER` /
`PAPER_DIR` into every step below.

## Step 1 — Extract the comparison surface from the ledger (decide whether to run)

Pull the claims this audit reasons over and decide if there is anything to audit:

```bash
ROOT=$(git rev-parse --show-toplevel 2>/dev/null || pwd)
LEDGER="<abs path to claims.json from Step 0>"
python3 - "$LEDGER" <<'PY'
import json, re, sys, collections
d = json.load(open(sys.argv[1], encoding="utf-8"))
claims = d.get("claims", [])
COMPARE = re.compile(r"\b(state[- ]of[- ]the[- ]art|SOTA|outperform\w*|best|"
                     r"surpass\w*|beats?|superior|first to|consistently|"
                     r"compared?\s+(?:to|with)|baseline|prior\s+(?:work|art))\b", re.I)
anchors, baselines = [], []
for c in claims:
    t, span = c.get("type"), c.get("text_span", "")
    # the SOTA / outperforms assertion (the anchor for completeness + fairness):
    if t == "comparison" or (t == "scope" and COMPARE.search(span)):
        anchors.append((c["claim_id"], t, c.get("location", {}).get("section", "?"), span[:150]))
    # the baseline SET the paper reports (mechanical; the reviewer decides completeness):
    if t == "baseline":
        baselines.append((c["claim_id"], c.get("location", {}).get("section", "?"), span[:150]))
print(f"ANCHOR (comparison/SOTA) claims: {len(anchors)}   baseline-list claims: {len(baselines)}")
for cid, t, sec, sp in anchors[:40]:
    print(f"  [anchor:{t}] {cid} [{sec}] {sp!r}")
for cid, sec, sp in baselines[:20]:
    print(f"  [baseline] {cid} [{sec}] {sp!r}")
nums = collections.Counter(c.get("type") for c in claims if c.get("type") in ("number", "table_cell"))
mets = collections.Counter((c.get("value") or {}).get("metric") for c in claims
                           if (c.get("value") or {}).get("metric"))
print("VALUE CLAIMS =", dict(nums), "  METRICS (seed the profile row) =", dict(mets))
print("APPLICABLE   =", "yes" if (anchors or baselines) else "no -> write [] and stop")
PY
```

**Branch.** If **APPLICABLE = no** (zero comparison/SOTA/baseline claims), this skill
is **not applicable**: write an empty `baseline-comparison-audit.findings.json`
(`[]`), record a one-line `NOT_APPLICABLE` reason in the trace (Step 7), and stop.
**Silent skip is forbidden** — the orchestrator globs `*.findings.json` and expects
the file to exist. Otherwise record the **anchor claims** (the SOTA/comparison
assertions) and the **reported-baseline list** for the prompts. A purely mechanical
grep helps surface the table/baseline names (do **not** judge completeness here — that
is the reviewer's job):

```bash
LEDGER="<abs path to claims.json from Step 0>"
grep -rInE '\\begin\{tabular|\\caption|baseline|w\.r\.t|vs\.?|\bours?\b' \
    "$(dirname "$LEDGER")" --include='*.tex' 2>/dev/null | head -60
```

## Step 2 — Assemble the candidate expected-baseline set (profile + live search + recency guard)

Determine the benchmark/task from the anchor claims (and the method section), then
build a **candidate expected set with sources** — structured *evidence*, not a
verdict. The recency guard is what stops you from naming a hallucinated or concurrent
baseline as "missing":

1. **Profile lookup.** If the task is in the per-domain profile above, take its
   expected baseline families (incl. the **floor**) and the matched-budget axis. If
   no row matches → mark the domain `NO_PROFILE`; completeness defaults to
   `needs_external_check`.
2. **Live leaderboard cross-check** (the authoritative source; record the query + the
   top systems + their **dates/venues**):
   ```
   WebSearch: "<benchmark> state-of-the-art <paper/current year> leaderboard"
   WebSearch: "<benchmark> papers with code"
   WebFetch:  <the Papers-with-Code / leaderboard URL>   # top 5–8 systems + their dates
   ```
3. **Recency + existence guard.** For each candidate baseline, record its
   `same_benchmark?` · `published_before_paper?` · `source_url` · `date`. A system
   concurrent with or post-dating the audited paper is a **legitimate omission** (a
   false positive for "missing"), not a flag.

Create the run's trace dir **now** — its first use is the file written just below, so
it must exist before Step 7. Reuse this exact `RUNDIR` in Steps 3–7 (do **not** create
a second one):

```bash
DATE=$(date +%Y-%m-%d); N=1
while [ -d ".aris/traces/baseline-comparison-audit/${DATE}_run$(printf %02d $N)" ]; do N=$((N+1)); done
RUNDIR=".aris/traces/baseline-comparison-audit/${DATE}_run$(printf %02d $N)"; mkdir -p "$RUNDIR"
echo "RUNDIR = $RUNDIR"   # carry this exact path forward (shell state does not persist)
```

Write these facts (not opinions) into `$RUNDIR/expected_baseline_set.json`:

```json
{
  "task": "GSM8K grade-school math (LLM reasoning)",
  "profile_version": "0.1",
  "profile_hit": true,
  "leaderboard_source": "https://paperswithcode.com/sota/arithmetic-reasoning-on-gsm8k  (read <UTC date>)",
  "expected_baselines": [
    {"name": "Self-Consistency CoT", "year": 2023, "same_benchmark": true, "published_before_paper": true, "source": "<url>"},
    {"name": "<recent frontier model> few-shot CoT", "year": 2024, "same_benchmark": true, "published_before_paper": true, "source": "<url>"}
  ],
  "notes": "off-profile or inconclusive search -> completeness becomes needs_external_check, not a guess"
}
```

**Failure handling.** No network / search fails → fall back to the profile alone and
mark every completeness candidate `requires_external_check: true` (you could not
confirm recency/availability). An empty candidate set + `NO_PROFILE` ⇒ skip the
completeness flag and emit a single `needs_external_check` info finding instead.
**Never** phrase the set as "the paper is missing X" — that is the reviewer's call.

## Step 3 — Completeness pass (cross-model, fresh thread) → HP-MISSING-BASELINE

Open a **fresh** `mcp__codex__codex` thread (Reviewer Calling Convention). The
reviewer reads `claims.json` from its `cwd` for the *present* baselines and compares
against your *external* expected-set facts; every finding anchors to a ledger
`claim_id`. Send EXACTLY (fill every `[ ... ]`):

```
mcp__codex__codex:
  model: gpt-5.5
  config: {"model_reasoning_effort": "xhigh"}
  sandbox: read-only
  cwd: <absolute PAPER_DIR from Step 0>
  prompt: |
    You are a baseline-COMPLETENESS forensics reviewer. You judge ONE thing: given
    what this paper claims ("state-of-the-art / best / first / outperforms prior
    work") on a benchmark, is an OBVIOUS, RECENT, RELEVANT baseline absent from the
    comparison? You do NOT judge whether numbers are real and you do NOT grade the
    paper. Describe a discrepancy to CHECK, never an accusation; hand off what you
    cannot ground.

    INPUTS (in your working directory — read them directly):
      - claims.json — the evidence ledger. The SOTA/comparison LANGUAGE lives in
        type:"comparison" and type:"scope" claims; the PRESENT baseline set in
        type:"baseline" claims; reported values + table rows in type:"number" /
        type:"table_cell"; figure/table labels in type:"caption". This is the ONLY
        structure you reason over; each claim = {claim_id, type, text_span (VERBATIM),
        location, value?}. You MAY re-open a source file to confirm a span is real,
        but you may NOT introduce a claim that is not in the ledger.
    REFERENCE (external facts gathered by the executor — cite the source in your
    finding; treat as GIVEN data, NOT a verdict, do not second-guess):
      - Profile row (PROFILE_VERSION 0.1): expected baseline FAMILIES = [...];
        matched-budget axis = [...].
      - Live leaderboard (<URL>, accessed <DATE>): top current systems + their
        dates/venues = [...].
      - Candidate expected set (name · same_benchmark? · published_before_paper? ·
        source): [paste expected_baseline_set.json from Step 2, or "NO_PROFILE /
        search unavailable"].
    ANCHOR TARGETS (the SOTA/comparison claims — claim_id + verbatim):
      [paste the anchor claims from Step 1]
    BASELINES THE PAPER REPORTS (mechanical extraction — may be incomplete):
      [paste the baseline-list claims + the grep names from Step 1]
    RUN OBSERVABILITY LEVEL L = <L from Step 0>.

    HARD RULES (a finding that breaks any of these is worthless):
    1. ANCHOR. Every finding above "info" MUST carry >=1 evidence {claim_id, span}
       where claim_id EXISTS in claims.json and span is a VERBATIM whitespace-
       normalized SUBSTRING of THAT claim's text_span (no paraphrase). The primary
       anchor is the SOTA/outperforms claim (a comparison/scope claim); the named
       missing baseline goes in `description`, never as the anchor. ALWAYS anchor —
       even a needs_external_check finding — so it stays navigable.
    2. DISCREPANCY, NOT ACCUSATION. Never "reject", "fabricated", "the authors hid X".
    3. OBSERVABILITY. A missing-baseline-as-STATED is decidable from the manuscript
       => observability_level_required = 0.
    4. HAND OFF WHAT YOU CANNOT SETTLE (the core rule of this step). Emit
       HP-MISSING-BASELINE above info ONLY when ALL hold: (a) a SOTA/best/outperforms
       claim is anchored; (b) a SPECIFIC, NAMED baseline is absent from the present
       set; (c) per the supplied sources that baseline is an ESTABLISHED, publicly-
       available, PRE-DATING standard for THIS exact benchmark (not concurrent, not
       post-dating, not unavailable, not justified-as-omitted in the paper). If ANY of
       (b)/(c) is uncertain — concurrent/post-dating per the dates, no code, niche
       benchmark, NO_PROFILE, the paper justifies the omission — DO NOT flag: set
       verdict_local "needs_external_check", requires_external_check true, severity
       "info", false_positive_risk "high", and name what a human should verify.
    5. HONEST FP. Concurrency/post-dating, unavailability, and a stated justification
       are the common false positives here — say so. If the expected set is empty/
       inconclusive, do NOT manufacture a missing baseline.
    6. pattern_id MUST be HP-MISSING-BASELINE.

    SEVERITY DECISION (HP-MISSING-BASELINE):
      - unambiguous, sourced, same-benchmark, clearly PRE-DATING omission AND the
        paper's HEADLINE is the SOTA/best claim -> "critical", FP "low",
        requires_external_check false.
      - clearly relevant + likely prior but contestable, OR a skipped strong FLOOR
        (BM25/GBDT/linear) while only weak baselines are beaten -> "major", FP
        "medium", requires_external_check true.
      - uncertain / NO_PROFILE / cannot confirm recency or relevance -> "info",
        verdict_local "needs_external_check", requires_external_check true.

    OUTPUT: a single JSON array and NOTHING ELSE (no prose, no code fence). Each
    element conforms to schemas/finding.schema.json:
      {"finding_id":"BC001","skill":"baseline-comparison-audit",
       "pattern_id":"HP-MISSING-BASELINE","title":"short, neutral",
       "description":"which expected baseline is absent + the SOTA claim it undermines + the source",
       "severity":"critical|major|minor|info","observability_level_required":0,
       "evidence":[{"claim_id":"C0xx","span":"verbatim substring",
                    "location":{"file":"...","section":"..."}}],
       "verdict_local":"fail|warn|clean|needs_external_check",
       "requires_external_check":true|false,"false_positive_risk":"low|medium|high",
       "recommended_reviewer_action":"what to CHECK or ASK — never 'reject'"}
    An empty array [] is a valid, honest result (the expected baselines are all present).
```

Persist the raw response to the trace dir (Step 7) **before** parsing. **Failure
handling:** MCP stall → re-invoke the **identical** prompt as a fresh
`mcp__codex__codex` (never `codex-reply`). Prose instead of JSON → the Step 5
validator extracts the outermost `[...]`; if none, re-ask once "Output ONLY the JSON
array." If the web cross-check was unavailable, bias every completeness item toward
`needs_external_check` — never invent a leaderboard entry.

## Step 4 — Fairness + Significance + cross-row Delta pass (cross-model, fresh thread; L2 deepen)

A **separate, new** `mcp__codex__codex` thread for the head-to-head rows. At **L2**
first gather mechanical config/seed facts (paths + raw grep/hash only — no
interpretation, the same executor/reviewer division as `experiment-forensics`);
**skip this block at L0/L1** (there are no configs to read):

```bash
ROOT=$(git rev-parse --show-toplevel 2>/dev/null || pwd)
PAPER_DIR="<abs PAPER_DIR from Step 0>"
L="<L from Step 0>"
# (L2 ONLY) per-method budget / tuning / seed asymmetry FACTS — paths + raw hits.
# Guarded: skip entirely at L0/L1 (no configs to read — the reviewer block gets "L<2: ...").
if [ "$L" = "2" ]; then
  grep -rInE 'epochs?|max_steps|learning_rate|\blr\b|batch_?size|seed|n_seeds|num_runs|backbone|warm.?up|sweep|tune|budget' \
      "$PAPER_DIR" --include='*.yaml' --include='*.yml' --include='*.json' --include='*.toml' 2>/dev/null | head -80
  find "$PAPER_DIR" -type f \( -name '*config*' -o -name '*args*' -o -name 'hparams*' \) 2>/dev/null | sort | head -40
  # space-safe and tolerant of zero matches (a bare `shasum $(...)` would hang on no args):
  find "$PAPER_DIR" -maxdepth 3 \( -name '*config*' -o -path '*results*.json' \) 2>/dev/null \
      | head -n 20 | while IFS= read -r ff; do shasum -a 256 "$ff" 2>/dev/null; done
fi
```

Then send EXACTLY (fill every `[ ... ]`):

```
mcp__codex__codex:
  model: gpt-5.5
  config: {"model_reasoning_effort": "xhigh"}
  sandbox: read-only
  cwd: <absolute PAPER_DIR from Step 0>
  prompt: |
    You are a baseline-FAIRNESS-and-SIGNIFICANCE forensics reviewer. For each head-
    to-head comparison the paper draws ("we outperform / are better than / achieve X
    vs baseline Y"), check whether it is FAIR (matched budget/tuning/config),
    SIGNIFICANT (the gap exceeds reported noise), and whether the stated cross-row
    improvement matches its operands. PROPOSE findings only — do NOT grade the paper;
    describe a discrepancy to CHECK, never an accusation. You do NOT judge whether the
    numbers are real (that needs the code) — only what the paper's OWN comparison shows.

    INPUTS (read directly in your working directory):
      - claims.json — comparison LANGUAGE in type:"comparison" / type:"scope"; the
        baseline set in type:"baseline"; values + table rows in type:"number" /
        type:"table_cell"; labels in type:"caption". The ONLY structure you navigate
        by; each claim = {claim_id, type, text_span (VERBATIM), location, value?}. You
        MAY re-open a source file (and, at L2, the config/result files) to confirm a span.
    RUN OBSERVABILITY LEVEL L = <L from Step 0>.
    HEAD-TO-HEAD comparison claims (anchor targets — claim_id + verbatim text_span):
      [paste the comparison/scope anchor claims + the relevant number/table_cell claims]
    REPORTED VARIANCE / SEEDS (if any, with claim_id):
      [paste any "+/-", "std", "over N seeds", "n=" spans, or write "NONE REPORTED"]
    L2 CONFIG FACTS (raw grep/hash hits + config paths — uninterpreted; empty if L<2):
      [paste the grep/find/shasum output above, or "L<2: no repo/configs available"]

    HARD RULES:
    1. ANCHOR every finding above "info" to a real claim_id + a VERBATIM substring of
       that claim. Anchor to the comparison claim (and the row's number/table_cell as
       extra evidence). A config file:line is forensic detail for the description —
       NOT a valid anchor. No verbatim span ⇒ keep at "info".
    2. DISCREPANCY, NOT ACCUSATION. Say what to CHECK/ASK. Never "reject"/"faked".
    3. OBSERVABILITY — set observability_level_required to the LOWEST tier at which the
       discrepancy is DECIDABLE: 0 when the asymmetry/variance/delta is VISIBLE IN THE
       PAPER TEXT/TABLES (e.g. "we train ours 300 epochs" vs a cited 90-epoch baseline
       number); 2 when CONFIRMING it needs the repo's config/result files (it auto-
       demotes on an L0/L1 run — that is correct, not a loss).
    4. HONEST FP. A documented identical budget, standard reference numbers cited from
       a baseline's own paper, a large gap, a reported significance test, and
       genuinely deterministic metrics are COMMON false positives — say so.
    5. pattern_id MUST be one of: HP-WEAK-BASELINE, HP-SIG-OVERLAP, HP-DELTA-ERROR,
       HP-RESOURCE-IDENTITY-MISMATCH.

    CHECKLIST (one finding per concrete discrepancy):
     1. FAIRNESS / WEAK BASELINE [HP-WEAK-BASELINE] — the proposed method gets more
        compute / tuning / data, or runs at more favorable settings, than the baseline;
        the compared rows use non-matching configs (backbone / data / split /
        decoding); a baseline is left at defaults while the method is tuned; a baseline
        number is copied from an old paper at a different budget. The strongest single
        check: is the method's OWN backbone-without-the-new-component (the ablation-as-
        baseline) reported at an IDENTICAL budget? severity major. observability 0 if
        the asymmetry is STATED in text; 2 if only the configs reveal it. FP: identical
        budget documented; a standard reference number quoted from the baseline's own
        paper (citing a published number is legitimate — note any config delta, do not
        allege).
     2. SIGNIFICANCE / OVERLAP [HP-SIG-OVERLAP] — "outperforms / better / consistently"
        claimed where reported error bars OVERLAP, or where NO variance / seed count is
        reported for a SMALL gap (within the field's typical noise for this metric).
        severity: major if error bars are reported AND overlap; minor if merely absent
        variance for a small gap (rise to major only if the headline rests on it).
        observability 0. FP (high → say so): a large gap; a significance test reported;
        a genuinely deterministic metric (exact match on a fixed test set). Two further
        thin-evidence signals — each a recurring real-review tell — to flag EXPLICITLY
        (SAME pattern_id HP-SIG-OVERLAP, SAME anchor = the comparison/SOTA claim):
        (a) NO VARIANCE / SEEDS REPORTED — the comparison rests on bare point estimates
            with NO ±/std/CI and NO seed/run count reported AT ALL (one number per cell,
            no "over N seeds"), so the gap cannot be told from run-to-run noise. severity
            minor for a small gap; major when the headline rests on it OR the profile's
            "Variance norm" column expects >=N seeds for this domain (e.g. RL >=5 seeds,
            retrieval per-query CI) and none are reported. observability 0. FP (high →
            say so): a large clearly-separated gap; a reported significance test; a
            genuinely deterministic / single-pass metric where seeds are moot.
        (b) SINGLE-DATASET-ONLY — a "consistently / robustly / across-the-board /
            general" comparison claim that rests on ONE dataset/benchmark (or a single
            split/domain), too thin for the breadth the wording asserts. severity minor;
            major only when that breadth IS the headline. observability 0. FP (high →
            say so): the claim is explicitly SCOPED to that one benchmark ("on GSM8K we
            …"); the dataset is the field-standard SOLE benchmark for the task; or broad
            scope genuinely exists elsewhere in the paper. LANE: this fires ONLY when
            anchored to a comparison/SOTA claim — generic scope-language inflation with
            NO comparison claim ("a comprehensive study") is consistency-audit's
            HP-SCOPE-INFLATE, not this; do NOT double-emit.
     3. CROSS-ROW DELTA ARITHMETIC [HP-DELTA-ERROR] — recompute a stated "improves over
        <baseline> by X%" as (proposed-baseline)/baseline AND absolute points; flag if
        X disagrees beyond rounding, or relative/absolute are conflated to inflate.
        severity major; critical if the corrected delta deflates a "large/significant"
        framing. observability 0. FP: abs-vs-rel stated explicitly; rounding. DELTA
        SCOPE (critical to avoid double-counting): flag ONLY when the two operands are
        a BASELINE row value and the PROPOSED row value living in DIFFERENT
        sentences/cells — the cross-row case a single-sentence regex cannot pair. Do
        NOT re-flag a delta whose operands AND stated value sit in ONE sentence — the
        deterministic consistency pass already owns those.
     4. RESOURCE IDENTITY [HP-RESOURCE-IDENTITY-MISMATCH] — a named dataset / benchmark /
        model is described with a checkable PUBLIC-RECORD property its registry contradicts
        (ImageNet-1k stated with the wrong #classes/size; a model's parameter count off from
        its card; a "SOTA 91.2 on <benchmark>" disagreeing with that benchmark's public
        leaderboard). RESOLVE each named resource against its HuggingFace dataset/model card
        or Papers-with-Code record (WebFetch/WebSearch — FACTS only; put the URL + access
        date in `description`); the reviewer judges the discrepancy. Anchor to the paper
        claim NAMING the resource (never the registry URL). severity major; critical if the
        mis-described resource IS the headline (the SOTA number that is the contribution).
        observability 0 (public-record contradiction) / 2 (the repo loads a different
        resource than named). FP (→ needs_external_check, never a guessed "wrong"): a
        declared subset/variant (ImageNet-100, a 10% split, a distilled/quantized model); a
        version difference (-21k vs -1k, v1 vs v2); an explicit redefinition; a stale /
        ambiguous registry or a leaderboard updated after submission. LANE: method
        described ≠ method evaluated is HP-METHOD-DRIFT (consistency-audit); a fabricated
        citation identity is HP-CITE-HALLUC — this is the named RESOURCE's identity vs its
        public record.

    OUTPUT: a single JSON array and NOTHING ELSE (schemas/finding.schema.json), same
    shape as the completeness prompt; set finding_id "BC0xx",
    skill "baseline-comparison-audit". An empty array [] is valid and honest. Set
    requires_external_check only when you genuinely cannot settle a point at this level.
```

**Deepen at L2.** When `L == 2`, the L2 config facts let the reviewer promote a
text-only suspicion to a confirmed `HP-WEAK-BASELINE` (keep `observability_level_required:
0` if the text already showed the asymmetry; use `2` for asymmetry only the configs
reveal). **Fan-out (optional, breadth).** On `— effort: max` or many comparison rows,
issue this checklist **per comparison entry** as a separate fresh `mcp__codex__codex`
call and concatenate the arrays — never `codex-reply`. Persist each raw reply to the
trace (Step 7). **Failure handling:** identical to Step 3 (fresh identical re-invoke
on stall; re-ask for the strict JSON array on prose; never hand-author findings).

## Step 5 — Validate + anchor + dedup (the anti-hallucination gate)

The executor enforces the **ANCHOR** gate (the one `tools/adjudicate_findings.py`
re-applies, so an anchored finding you keep is not silently rejected downstream) plus
baseline-specific **owned-pattern + delta-dedup + schema-hygiene** pre-filters,
**before** keeping anything. The span must be a verbatim, whitespace-normalized
**substring of** the cited claim (`span in base`, never `base in span` — appending
hallucinated text to a real claim must fail). Pass **every** saved raw reviewer
response (completeness + fairness + any per-entry fan-out files); they merge into one
findings file with one `BC###` namespace:

```bash
ROOT=$(git rev-parse --show-toplevel 2>/dev/null || pwd)
LEDGER="<abs path to claims.json>"
OUT="$(dirname "$LEDGER")/baseline-comparison-audit.findings.json"
# args: LEDGER OUT then each saved raw reviewer response file from Steps 3–4:
python3 - "$LEDGER" "$OUT" "<resp_completeness.md>" "<resp_fairness.md>" <<'PY'
import json, re, sys, os
ledger_path, out_path = sys.argv[1], sys.argv[2]
resp_paths = [p for p in sys.argv[3:] if p and os.path.isfile(p)]

def nw(s):                      

…(truncated)