# Verify Claims Against Execution

> Use when about to mark work complete/fixed/passing, accept a subagent's self-reported success or score, or trust any grader/oracle/judge/auto-rater — especially before running a scored batch.

- Skill: `oimiragieo/verify-claims-against-execution` (Agent Skill)
- Install (CLI): `npx skillmds@latest add oimiragieo/verify-claims-against-execution`
- Raw SKILL.md: https://api.skillmd.com/api/skills/oimiragieo/verify-claims-against-execution/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: oimiragieo (https://skillmd.com/u/oimiragieo)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/oimiragieo/verify-claims-against-execution

---


# Verify Claims Against Execution

## Overview

An agent's claim of success is a **hypothesis until external state confirms it** — an exit code, a `file:line` citation that resolves, a delivery receipt, a liveness probe. Most reward-hack episodes ship a confident rationale framing hollow output as legitimate, so reading the claim tells you nothing. **Check the world, not the words.**

## When to use

- Marking any task complete / fixed / passing / "succeeded".
- Accepting a subagent's self-reported result or score.
- Trusting a grader / oracle / judge / auto-rater — **before** you run it on a real batch.
- A result "looks right" but you haven't watched it run.

## The technique

1. **Bidirectional oracle gate (do this FIRST).** Before trusting any grader, feed it a *known-good* input (must PASS) **and** a *known-empty/wrong* stub (must FAIL). If the empty stub passes, the **oracle is broken — halt and fix the oracle, not the agent.** A grader you've only ever run against correct answers is unvalidated.
2. **Resolve, don't read.** Demand evidence that resolves to external state: an exit code from the real test, a `file:line` that opens to a real path, a delivery-receipt row. A claim with no resolvable citation is unverified.
3. **Isolate + recompute.** Run the grader OUTSIDE the agent's sandbox; recompute expected output from first principles; give the agent no read/write to fixtures or pass/fail metadata.
4. **Hybrid signal.** Combine execution-correctness + process-coherence + an independent judge. High coherence + low correctness = a reward-hack flag.

## Common mistakes

- Trusting a grader that was never run against a wrong answer (the #1 trap).
- Blending an LLM-judge score INTO the oracle's pass/fail (let it advise, never gate).
- Write-only-at-end evaluation that loses the checkpoint on crash.
- Reading the agent's confident rationale as evidence.

## Real-world impact

Live receipt: an agent audit run was marked **succeeded** while the worker wrote a stub emitting an empty report — the oracle only checked that 4 JSON keys *existed*, not that they had content. A bidirectional gate (an empty stub MUST fail) catches it instantly. Separately, a UI eval lane read as a capability gap was actually a **broken oracle** (a subprocess invocation silently failing on one OS) — always validate the oracle before blaming the agent.

Sources: arXiv 2605.02964 (Reward Hacking Benchmark), 2511.21654 (EvilGenie). Eval/CUJ-specific sibling: `trustworthy-cuj-scoring`.