Eval Integrity

eval-integrity — Benchmark Credibility Audit

User-invokable auditor for LLM evaluation and benchmark repos. It answers one question: if you published this benchmark's numbers, would they survive an adversarial reviewer?

Most benchmark repos have a runner and a leaderboard but lack the integrity scaffolding that makes a published score mean what it appears to mean. This skill checks for that scaffolding, names what is missing, and rates whether each gap invalidates published numbers or is nice-to-have.

The checks below were extracted from hardening a real agent benchmark (cot-bench, a private podcast-evals repo) for an external grant review. Where a check needs a concrete example, cot-bench file names appear as illustrations — they are hints for what good looks like, not paths the auditor expects to find in your repo.

This is the static-analysis cousin of code-review: same parallel-subagent orchestration, but the subagents audit eval methodology instead of code correctness.

Invocation: deliberately model-invocable — read-only audit; it reports and offers fixes, never edits the benchmark.

When to Use

• Before submitting a benchmark to a grant, conference, or public leaderboard.
• When someone says "I don't trust those numbers" and you need to know if they're right.
• After building an eval, before publishing the first headline result.
• Periodically, as a benchmark accretes models and the leaderboard becomes load-bearing.

When NOT to Use

• A repo that is not an eval or benchmark (no scoring, no leaderboard, no judge). This skill has nothing to grade.
• A toy eval with no published numbers and no intent to publish. Integrity scaffolding is overhead until someone relies on the score.

Step 1: Confirm the target is a benchmark and locate its parts

Target: $ARGUMENTS (path to the benchmark repo; if empty, assume the current directory).

Establish the repo is gradable, then map where the audit will look. Run from the target repo root:

# Scoring / judging present?
rg -l --hidden -g '!.git' "judge|rubric|score|grade|leaderboard|eval" .
# Where do results / leaderboards live?
rg -l "leaderboard|results|latest\.csv|\.parquet" .
# Is there a stated methodology or governance doc?
rg -l -i "methodology|governance|contamination|pre.?registration" .

If none of these hit, the repo is probably not a benchmark. Stop and say so.

Record for the subagents:

• Absolute repo path.
• Branch + HEAD commit SHA (git rev-parse HEAD).
• The scoring entry point(s), the results directory, and any methodology/governance docs.
• The target's open PRs (gh pr list). This is a standing step, not optional. Verify any relevant PR against its actual diff. A finding that an in-flight PR already fixes is reported as known/in-flight, not as a gap, and is excluded from the gap counts.

Step 2: Spawn one subagent per dimension, in parallel

Spawn the seven dimension auditors in a single tool-call batch so they run in parallel. Each gets the repo path, HEAD SHA, the located parts from Step 1, and its dimension's audit brief. The paste-ready briefs live in patterns/dimension-prompts.md — one per dimension, each already carrying its grep patterns, severity rule, and report contract.

The seven dimensions:

Each subagent prompt must specify:

1. Repo path (absolute), branch, HEAD SHA.
2. The located scoring/results/docs from Step 1.
3. The dimension brief from patterns/dimension-prompts.md.
4. The exact report contract (Step 3 format), capped at ~400 words.

If the repo is small (one scoring file, no CI, no leaderboard), you may run the dimensions inline yourself instead of spawning subagents. Parallel subagents pay off on a real benchmark with CI, a results pipeline, and a methodology doc.

If the environment has no subagent or Task tool, do not skip or thin the audit. Run the dimension briefs inline yourself, one after another, in dimension order. Same briefs, same evidence bar, same report contract. Note in the report that the audit ran sequentially with a single auditor.

Step 3: Scoring rule each subagent applies

Every dimension returns one rating, with evidence:

• PRESENT — the practice is implemented AND enforced (code or test, not just prose). Cite the file:line.
• PARTIAL — documented or half-built. The intent exists; the enforcement does not (e.g. governance.md promises a holdout gap, but no code computes it; a comment says "authors excluded" but no guard blocks it).
• ABSENT — no evidence in code, tests, or docs.

Evidence is mandatory. A rating with no file:line (or an explicit "searched X, Y, Z — found nothing") is not a finding, it's a guess. Reject it.

A grep miss alone does not establish ABSENT. The briefs' grep patterns carry one repo's vocabulary; the target may name the same concept differently. Search by concept (read the scoring entry points, the CI workflows, the docs) before rating any sub-check ABSENT.

Step 4: Assign severity

Severity is about consequence, not effort. Two levels:

• INVALIDATING — the gap means a published number could be wrong, gamed, or non-comparable, and a reviewer who finds it can dismiss the result. Examples: no author-is-contestant guard (contamination), judge not pinned to the served model (silent drift), holdout content reachable in CI artifacts (leak), no CIs on a headline ranking built from few scenarios (noise sold as signal), null-agent or holdout rows in the public leaderboard with no tripwire test (gameability / overfitting hidden).
• HARDENING — the gap weakens credibility or auditability but does not by itself invalidate a number. Examples: agreement stats present but not chance-corrected (within-0.2 rate instead of Krippendorff's alpha), no resume/cost-cap on expensive runs, environment pinned in prose but not a lockfile, multiple-comparison risk unacknowledged in an otherwise CI'd board.

When unsure, ask: can a critic use this gap to throw out the headline number? If yes, INVALIDATING.

Step 5: Assemble the report

Collect the seven subagent reports into one scored audit. Lead with the verdict and the invalidating gaps — those are what the author needs to fix before publishing.

EVAL-INTEGRITY AUDIT — <repo> @ <short-sha> — <date>

VERDICT: <PUBLISH-READY | N INVALIDATING GAP(S) | NOT A BENCHMARK>
Score: <count PRESENT>/7 present, <count PARTIAL> partial, <count ABSENT> absent

INVALIDATING GAPS (fix before publishing)
- [<dimension>] <one line>. Evidence: <file:line or "absent: searched X,Y,Z">. Fix: <concrete change>.

HARDENING GAPS (raise credibility)
- [<dimension>] <one line>. Evidence: <…>. Fix: <…>.

PER-DIMENSION
1. Pre-registration    — PRESENT | PARTIAL | ABSENT  — <evidence> — <fix if not PRESENT>
2. Contamination       — …
3. Holdout hygiene     — …
4. Judge validity      — …
5. Statistical honesty — …
6. Reproducibility     — …
7. Leaderboard excl.   — …

STRENGTHS (what's already solid — one line each)
- …

Rules for the report:

• Lead with what invalidates. Hardening gaps go below. Strengths last.
• Every gap carries a concrete fix, not "consider improving X." Name the file to add, the guard to write, the test to add. If cot-bench solves it a particular way, that pattern is a fair suggested fix.
• No fix is "add more scenarios" unless low scenario count is the specific finding (e.g. CIs so wide the ranking is noise).
• In-flight fixes are not gaps. If an open PR on the target already fixes a finding (verified against its diff, not its title), list it under a KNOWN / IN-FLIGHT line after the hardening gaps, name the PR, and leave it out of the gap counts.
• Plain and direct. Short sentences. No hype. State the gap, the evidence, the fix.

Step 6: Offer the fixes (do not auto-apply)

Auditing is read-only. After the report, ask which gaps to fix. Applying a fix changes the target repo's methodology — that is the author's call, one gap at a time, with their approval. Do not edit the benchmark, re-run any eval, or touch a leaderboard as part of the audit.

Why these seven

Each dimension maps to a way a benchmark's numbers get dismissed in review:

• Pre-registration defeats run cherry-picking ("you re-ran until it looked good").
• Contamination defeats "the author's model graded the author's exam."
• Holdout hygiene defeats "your private set isn't private — it's in the CI logs."
• Judge validity defeats "your judge drifted / favors its own family / rewards length."
• Statistical honesty defeats "that ranking is sampling noise."
• Reproducibility defeats "no one can re-run this."
• Leaderboard exclusions defeats "a do-nothing agent ranks mid-board" and "you trained on what you published." Its publish-mechanics sub-check defeats "the table your docs promise is never actually written."

A benchmark that passes all seven is one whose headline number a skeptical reviewer has to take seriously.

Cross-references

• patterns/dimension-prompts.md — paste-ready audit brief per dimension, with grep patterns, severity rule, and report contract.