# Independent Oracle Validation

> Scores committed extracts against an independent extraction engine (a different codebase than the pipeline lane) to produce measured per-lane precision/recall, with two-tier numeric scoring and a hard rule that every sub-1.0 number is attributed to a named cause. Use when a conversion lane needs correctness evidence beyond the fidelity verify loop — turning "we believe the extracts are faithful" into per-lane measured numbers.

- Skill: `vamseeachanta/independent-oracle-validation` (Agent Skill, multi-file: 2 files)
- Install (CLI): `npx skillmds@latest add vamseeachanta/independent-oracle-validation`
- Raw SKILL.md: https://api.skillmd.com/api/skills/vamseeachanta/independent-oracle-validation/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: DevOps & Infra
- License: CC-BY-4.0
- Author: vamseeachanta (https://skillmd.com/u/vamseeachanta)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/vamseeachanta/independent-oracle-validation

---


# independent-oracle-validation

> Template skill (doc 03, doc 05 GP-47). The per-batch verify loop proves
> **fidelity** — the extract matches what the pipeline read. It cannot prove
> **correctness**: the pipeline and its verifier can share a defect. This
> skill adds the missing leg: re-extract the same sources with a *different
> engine* and score the committed extracts against it. In the pilot (11 docs,
> 6 office lanes) the measured numbers were 1.000/1.000 for data and deck
> lanes, with every sub-1.0 residual resolving to a named cause — including
> one previously unledgered real lane loss (render-time list numbering) that
> no amount of self-consistent verification would ever have surfaced.

> **The comparator is the trap.** The pilot's scoring tool needed FOUR
> adversarial review rounds — all on the comparator, none on the pipeline:
> its "exact" tier silently rounded numerics three different ways before
> canonicalization was precision-lossless *by construction*. A flattering
> comparator bug poisons every downstream number, so the scoring tool gets
> the verify loop at higher rigor than the lane it scores.

## Trigger
`/independent-oracle-validation [--out report.md]`

## Preconditions
1. **Oracles are a different codebase than the lane**, per format — e.g.
   office-suite headless conversion vs a python reader; a second PDF text
   engine; raw container-XML text nodes vs an object-model library; an
   alternative implementation of the same format. Where independence is
   partial (two libraries sharing a container parser), the report says so
   explicitly — shared layers prove less.
2. The committed extracts and the source binaries (or a hash-verified cache)
   are available; the corpus root comes from an env var, never a committed
   path.
3. The comparison is reproducible: same inputs → byte-identical report.

## Steps
1. **Re-extract with the oracle** per document, into a temp dir only (the
   raw-binary firewall applies to oracle conversions too).
2. **Normalize formatting, never precision.** Strip currency symbols,
   thousands separators, display percent signs, accounting negatives —
   transformations that are provably formatting-only. Numeric
   canonicalization must be precision-lossless at ANY digit count (decimal
   with width-matched context, not binary float, not bounded-precision
   normalize). Cell-only equivalences (accounting `-` = 0) never apply to
   prose tokens.
3. **Score two tiers** of multiset precision/recall per document: **exact**
   (formatting-normalized) and **display-tolerant** (e.g. 3 significant
   figures), because oracles that export *displayed* values legitimately
   disagree with lanes that store full precision. Report both; never let the
   tolerant tier masquerade as exact.
4. **Strip only your own scaffold** from the lane side (structural markers,
   provenance headers, additive inventories) — positionally, so look-alike
   content inside the document body is preserved. Scaffold is your addition;
   body content the oracle also sees is not.
5. **Attribute every sub-1.0 number.** Diff the bags; each residual class
   resolves to exactly one of: a **real lane loss** → record it in the
   coverage ledger; a **legitimate engine difference** → caveat it in the
   report; a **comparator bug** → fix the comparator and re-run. Never
   adjust a score without naming its cause.
6. **Run the adversarial verify loop on the comparator** (skill
   `adversarial-verify-loop`), instructing reviewers to hunt score-inflating
   normalization collapses with constructed probes. Loop until PASS.
7. **Publish the report** (tool versions pinned, caveats included) and cite
   the measured numbers from the lane-maturity matrix — scoped honestly:
   value/token overlap complements, never replaces, information-capture
   completeness estimates or ordering checks.

## Verification
- Re-running the tool reproduces the committed report byte-for-byte.
- Every sub-1.0 number in report and matrix has a named, probe-verified
  attribution.
- The comparator has its own PASS-terminated adversarial review record.
- Oracle conversions never wrote inside the repo tree.
- Published oracle reports and review evidence pass
  `bash scripts/legal/legal-sanity-scan.sh --all-tracked-public-surfaces`; use
  `--diff-only` before committing local report changes.

## Cleanup
- Oracle conversion outputs and probe scripts stay in temp/session dirs;
  only the report and the scoring tool are committed.

## Incident appendix
| Rule | Why |
|---|---|
| Oracle from a different codebase | Same-engine re-runs prove reproducibility, not correctness |
| State partial independence honestly | Two readers sharing a container parser prove less than two unrelated engines |
| Precision-lossless exact tier, by construction | Three successive "fixes" each handled only the probed case; bounded-precision contexts still rounded |
| Two tiers, both reported | Display-rounding is a legitimate engine difference; hiding it in "exact" inflates, splitting it attributes |
| Attribute every sub-1.0 number | One residual was a real, previously unledgered lane loss; the rest were engine formatting — only the diff probe tells them apart |
| Verify the comparator harder than the pipeline | A flattering comparator bug poisons every number it produces |
| Scaffold stripped positionally | Prefix-stripping by pattern ate body content that looked like scaffold (quoted-history subject lines) |