Experiment Audit

Experiment Audit: Cross-Model Integrity Verification

🔒 Do not wrap this skill in /loop, /schedule, or CronCreate. It is verdict-bearing — it judges experiment integrity. Re-running that verdict on a timer adds no new signal, and a loop that accepts its own output to decide when to stop crosses into self-acquittal (acceptance-gate.md). Schedule the external wait that precedes it — experiments done → then audit once. See shared-references/external-cadence.md.

Audit experiment integrity for: $ARGUMENTS

Why This Exists

LLM agents can produce fraudulent experimental results through:

1. Fake ground truth — creating synthetic "reference" from model outputs, then reporting high agreement as performance
2. Score normalization — dividing metrics by the model's own max to get 0.99+
3. Phantom results — claiming numbers from files that don't exist or functions never called
4. Insufficient scope — reporting 2-scene pilots as "comprehensive evaluation"

These are NOT intentional deception — they are failure modes of optimizing agents that lack integrity constraints. This skill adds that constraint.

Core Principle

The executor collects file paths. The reviewer — external when available, Claude itself otherwise — reads code and judges integrity, separately from the collection step. Even under self-review, keep these as two distinct passes: Step 1 collects paths without reading/summarizing content; the review step (below) is the first time the content itself gets read and judged, and it re-reads fresh from disk rather than from any impression formed while listing files.

This follows shared-references/reviewer-independence.md and shared-references/experiment-integrity.md. See auto-review-loop's Self-Review Backend for the general self-review rationale and its honest tradeoff: no independence guarantee, and it cannot catch a fraud pattern Claude is systematically blind to the way a different model could.

Constants

• REVIEWER_BACKEND = codex — Default: Codex MCP (ultra). Override with — reviewer: oracle-pro for Oracle MCP, — reviewer: manual for Manual Review MCP, or — reviewer: self when no external model/API is configured at all. If neither Codex, Oracle, nor manual-review MCP is available, fall back to self rather than stopping — integrity auditing is this skill's entire purpose, so silently skipping it is worse than a self-review with a clearly labeled independence caveat. See shared-references/reviewer-routing.md.

Reviewer Calling Convention

When calling the reviewer, branch on REVIEWER_BACKEND:

If REVIEWER_BACKEND = codex: Use mcp__codex__codex for new review threads. Use mcp__codex__codex-reply for follow-up rounds (reuse threadId).

If REVIEWER_BACKEND = manual: Use mcp__manual_review__review for new review threads with: prompt: [exact same prompt that would go to Codex] config: {"model_reasoning_effort": "xhigh", "executor_model": "", "require_reviewer_model": true} Save the returned threadId. Use mcp__manual_review__review_reply for follow-up rounds with: threadId: [saved manual-review threadId] prompt: [follow-up prompt] config: {"model_reasoning_effort": "xhigh", "executor_model": "", "require_reviewer_model": true}

If REVIEWER_BACKEND = self: No second model is available. Read every file collected in Step 1 directly — the same files, the same checklist — as an adversarial auditor who assumes the evaluation is compromised somewhere. Follow auto-review-loop's "Self-Review Backend (No Second Model)" for the full protocol: re-read from disk (not from memory of implementing the evaluation), argue against the results you'd like to be true, and flag any check you cannot judge impartially. Record identity_assurance: "self_review_no_independence" and independence_verified: false in EXPERIMENT_AUDIT.json instead of a Reviewer-Model: header.

Prompt fidelity: the manual prompt must be exactly the same text that Codex would receive; the self route uses the same checklist as its own instructions. Review tracing applies equally to all backends.

Workflow

Step 1: Collect Artifacts (Executor — Claude)

Locate and list these files WITHOUT reading or summarizing their content:

Scan project directory for:
1. Evaluation scripts:    *eval*.py, *metric*.py, *test*.py, *benchmark*.py
2. Result files:          *.json, *.csv in results/, outputs/, logs/
3. Ground truth paths:    look in eval scripts for data loading (dataset paths, GT references)
4. Experiment tracker:    EXPERIMENT_TRACKER.md, EXPERIMENT_LOG.md
5. Paper claims:          NARRATIVE_REPORT.md, paper/sections/*.tex, PAPER_PLAN.md
6. Config files:          *.yaml, *.toml, *.json configs with metric definitions

A verdict-bearing manual response MUST begin with Reviewer-Model: <exact-model-id> — pass the model THIS session is actually running as in executor_model. Missing, unknown, or same-family identity cannot acquit; emit REVIEW_UNAVAILABLE rather than guessing. If the executor model cannot be named, manual review's cross-family claim is unprovable — say so in the report instead of asserting it.

DO NOT summarize, interpret, or explain any file content. Only collect paths.

Step 2: Send to Reviewer

Based on the selected reviewer backend (see Reviewer Calling Convention), pass ONLY file paths and the audit checklist to the reviewer. The reviewer reads everything directly.

For codex, call mcp__codex__codex with:

• model: gpt-5.6-sol
• config: {"model_reasoning_effort": "ultra"}
• sandbox: read-only
• cwd: [project directory]
• prompt: [the exact full prompt below]

For manual, call mcp__manual_review__review with:

• config: {"model_reasoning_effort": "xhigh", "executor_model": "<actual executor model>", "require_reviewer_model": true}
• prompt: [the exact full prompt below]

Manual review cannot use Codex-only model, sandbox, or cwd; include the same file paths in the prompt so the user can inspect them.

For self, there is no tool call: open the exact file paths from Step 1 directly and work through the checklist below yourself, fresh, per the Reviewer Calling Convention.

Use this exact prompt/checklist for all backends:

You are an experiment integrity auditor. Start from the assumption that the
    evaluation is compromised somewhere — your job is to find where. Be
    adversarial. Trust nothing the author tells you — verify everything
    yourself. Read ALL files listed below and check for the following fraud
    patterns.

    Files to read:
    - Evaluation scripts: [list paths]
    - Result files: [list paths]
    - Experiment tracker: [list paths]
    - Paper claims: [list paths]
    - Config files: [list paths]

    ## Audit Checklist

    ### A. Ground Truth Provenance
    For each evaluation script:
    1. Where does "ground truth" / "reference" / "target" come from?
    2. Is it loaded from the DATASET, or generated/derived from MODEL OUTPUTS?
    3. If derived: is it explicitly labeled as proxy evaluation?
    4. Are official eval scripts used when available for this benchmark?
    FAIL if: GT is derived from model outputs without explicit proxy labeling.

    ### B. Score Normalization
    For each metric computation:
    1. Is any metric divided by max/min/mean of the model's OWN output?
    2. Are raw scores reported alongside any normalized scores?
    3. Are any scores suspiciously close to 1.0 or 100%?
    FAIL if: Normalization denominator comes from prediction statistics.

    ### C. Result File Existence
    For each claim in the paper/narrative:
    1. Does the referenced result file actually exist?
    2. Does the claimed metric key exist in that file?
    3. Does the claimed NUMBER match what's in the file?
    4. Is the experiment tracker status DONE (not TODO/IN_PROGRESS)?
    FAIL if: Claimed results reference nonexistent files or mismatched numbers.

    ### D. Dead Code Detection
    For each metric function defined in eval scripts:
    1. Is it actually CALLED in any evaluation pipeline?
    2. Does its output appear in any result file?
    WARN if: Metric functions exist but are never called.

    ### E. Scope Assessment
    1. How many scenes/datasets/configurations were actually tested?
    2. How many seeds/runs per configuration?
    3. Does the paper use words like "comprehensive", "extensive", "robust"?
    4. Is the actual scope sufficient for those claims?
    WARN if: Scope language exceeds actual evidence.

    ### F. Evaluation Type Classification
    Classify each evaluation as:
    - real_gt: uses dataset-provided ground truth
    - synthetic_proxy: uses model-generated reference
    - self_supervised_proxy: no GT by design
    - simulation_only: simulated environment
    - human_eval: human judges

    ## Output Format

    For each check (A-F), report:
    - Status: PASS | WARN | FAIL
    - Evidence: exact file:line references
    - Details: what specifically was found

    Overall verdict: PASS | WARN | FAIL
    
    Be thorough. Read every eval script line by line.

Step 3: Parse and Write Report (Executor — Claude)

Parse the reviewer's response and write EXPERIMENT_AUDIT.md:

# Experiment Audit Report

**Date**: [today]
**Auditor**: [reviewer backend used — codex/manual (external, cross-model, read-only) or self (Claude, no second model available)], ultra reasoning
**Project**: [project name]

## Overall Verdict: [PASS | WARN | FAIL]

## Integrity Status: [pass | warn | fail]

## Checks

### A. Ground Truth Provenance: [PASS|WARN|FAIL]
[details + file:line evidence]

### B. Score Normalization: [PASS|WARN|FAIL]
[details]

### C. Result File Existence: [PASS|WARN|FAIL]
[details]

### D. Dead Code Detection: [PASS|WARN|FAIL]
[details]

### E. Scope Assessment: [PASS|WARN|FAIL]
[details]

### F. Evaluation Type: [real_gt | synthetic_proxy | ...]
[classification + evidence]

## Action Items
- [specific fixes if WARN or FAIL]

## Claim Impact
- Claim 1: [supported | needs qualifier | unsupported]
- Claim 2: ...

Also write EXPERIMENT_AUDIT.json for machine consumption:

{
  "date": "2026-04-10",
  "auditor": "external-reviewer-ultra | self-review-ultra",
  "reviewer_backend": "codex | manual | self",
  "independence_verified": true,
  "overall_verdict": "warn",
  "integrity_status": "warn",
  "checks": {
    "gt_provenance": {"status": "pass", "details": "..."},
    "score_normalization": {"status": "warn", "details": "..."},
    "result_existence": {"status": "pass", "details": "..."},
    "dead_code": {"status": "pass", "details": "..."},
    "scope": {"status": "warn", "details": "..."},
    "eval_type": "real_gt"
  },
  "claims": [
    {"id": "C1", "impact": "supported"},
    {"id": "C2", "impact": "needs_qualifier"}
  ]
}

Step 4: Print Summary

🔬 Experiment Audit Complete

  GT Provenance:      ✅ PASS — real dataset GT used
  Score Normalization: ⚠️ WARN — boundary metric uses self-reference
  Result Existence:    ✅ PASS — all files exist, numbers match
  Dead Code:           ✅ PASS — all metric functions called
  Scope:               ⚠️ WARN — 2 scenes, paper says "comprehensive"

  Overall: ⚠️ WARN
  
  See EXPERIMENT_AUDIT.md for details.

Integration with Other Skills

Automatic in /research-pipeline (advisory, never blocks)

When integrated into the pipeline, this skill runs automatically after /experiment-bridge and before /auto-review-loop:

/experiment-bridge → results ready
    ↓
/experiment-audit (automatic, advisory)
    ├── PASS  → continue normally
    ├── WARN  → print ⚠️ warning, continue, tag claims as [INTEGRITY: WARN]
    └── FAIL  → print 🔴 alert, continue, tag claims as [INTEGRITY CONCERN]
    ↓
/auto-review-loop → proceeds with integrity tags visible to reviewer

Never blocks the pipeline. Even on FAIL, the pipeline continues — but claims carry visible integrity tags.

Read by /result-to-claim (if exists)

if EXPERIMENT_AUDIT.json exists:
    read integrity_status
    attach to verdict: {claim_supported: "yes", integrity_status: "warn"}
    if integrity_status == "fail":
        downgrade verdict display: "yes [INTEGRITY CONCERN]"
else:
    verdict as normal, integrity_status = "unavailable"
    mark as "provisional — no integrity audit"

Read by /paper-write (if exists)

if EXPERIMENT_AUDIT.json exists AND integrity_status == "fail":
    add footnote to affected claims: "Note: integrity audit flagged concerns with this evaluation"

Key Rules

• Reviewer independence: executor collects paths, reviewer judges — as two distinct passes, even when both are Claude. Period.
• Never block: warn loudly, never halt the pipeline.
• File-as-switch: no EXPERIMENT_AUDIT.md = skill was never run = zero impact on existing behavior.
• Cross-model when possible: prefer codex/manual (a different model family from the executor) whenever one is available; fall back to self only when none is reachable, and record that in identity_assurance/independence_verified (see the Reviewer Calling Convention's self route).
• Honest about limits: the audit catches common patterns, not all possible fraud. It is a safety net, not a guarantee — and under self, it also lacks the independence a different model would provide (see auto-review-loop's Self-Review Backend).

Acknowledgements

Motivated by community-reported integrity issues (#57, #131) where executor agents created fake ground truth and self-normalized scores.

Review Tracing

After each reviewer call or self-review pass (mcp__codex__codex, mcp__codex__codex-reply, mcp__manual_review__review, mcp__manual_review__review_reply, or the self route), save the trace following shared-references/review-tracing.md (Policy C — forensic; never silently skip). Use save_trace.sh (resolved per the chain in shared-references/integration-contract.md §2) or write files directly to .aris/traces/<skill>/<date>_run<NN>/. Respect the --- trace: parameter (default: full).