vamseeachanta source-extract-fidelity
Template skill (doc 03, doc 04). Extraction can be byte-perfect and the page still wrong — because the prose around the extract claims more than the extract supports. Across a 53-document campaign the dominant defect class was not bad extraction but prose overclaiming beyond the extract: a number that was cross-referenced (not quoted), a value that was derived (not stated), or a clause re-ordered so it no longer matches the source. This skill is the independent verify loop that catches that.
What this gates: the batch PR (a
needs-verify→verifiedlabel flow), not each page'sparse_status. Pages may stayparse_status: provisionaleven after the batch passes — per-row promotion toverifiedis a separate step. This skill is the independent batch review run before the PR is cleared to merge.
Trigger
/source-extract-fidelity <batch> [--round 1|2]
Preconditions
- The reviewer is independent of the producer — a different agent/pass, prompted to find a reason the page is wrong, not to confirm it looks fine.
- The source binary is available as a verified copy (hash matches its pointer);
for off-disk LFS binaries, fetch via
lfs-batch-fetchfirst. - Each source's sha256 is recorded in the page's source table and in
sources/README.md, and pages use the closed status vocabulary (parse_status/trust_label/derivation_status).
Steps
- Hash gate. Recompute the source sha256 and confirm it matches the value
recorded in the page's source table /
sources/README.md. A source that can't be reproduced from its recorded hash is invalid before any prose is read. - Extract-fidelity check. Compare the committed extract (txt/CSV) against the
source: did the deterministic lane capture what it claims (right sheet/slide/
thread, no silent truncation)? Note any known-lossy gap (see
format-coverage-ledger) but do not penalize a faithful extract for the text/CSV lane's known loss. - Quote traceability — the core check. For every quoted figure, name, or
clause on the page, find the exact span in the committed extract. Apply the
closed rules:
- A cross-reference ("per the calc in sheet X") is NOT a quote — it must not be rendered as if the value were stated here.
- A derived number (computed/rounded/unit-converted from extract values)
is NOT "quoted" — it is
derivation_status: derived, never a verbatim capture. - A quote must match clause order exactly. Re-ordering clauses to read better changes meaning; a re-ordered quote is a defect, not a paraphrase.
- Overclaim sweep. Read every declarative sentence and ask: does the committed extract contain this? Anything beyond the extract (interpretation, conclusion, "this means…") must be labeled output/analysis, never presented as source. Unsupported source-claims → flag for correction. For ACE wave-1 config/code-doc rows, claims must trace to the retained metadata, docstring, comment, or schema key. Generated JSON exclusions must not be converted into prose claims about source content.
- Invariant + vocab check. Confirm raw binary is not committed (only derived parts + pointer), private content hasn't crossed to public, and every status value is in the closed set (no invented labels).
- Verdict + further rounds until PASS. Emit a per-page verdict with cited
defects. Each later round re-checks what the producer corrected, again
adversarially — a "fixed" page often just moved the overclaim, and a code
fix can be narrower than the defect class. Two rounds is the floor, not
the ceiling: the loop ends at a clean PASS, never at a round count (GP-42;
one campaign fix needed a third round, caught by a reviewer-built
reproducer — see
adversarial-verify-loop).
Verification
- Every quoted span on every page resolves to an exact location in the committed extract (the traceability map is complete — no orphan quotes).
- No derived/cross-referenced value is presented as a verbatim quote.
- Config-derived and code-doc claims trace to retained metadata/docstrings; no prose claim is inferred from excluded generated JSON.
- Hash matches; invariants and closed-set vocab hold.
- Defects cite a specific falsifiable location ("page says X, extract has Y"), not generic approval/disapproval.
Cleanup
- n/a (read-only review). The fetched source temp copy is removed by the fetch skill's cleanup.
Incident appendix
| Rule | Why |
|---|---|
| Independent, adversarial reviewer | Charitable self-review rubber-stamps; the producer can't see its own overclaim |
| Quote must trace to a committed extract | The #1 defect class: prose claiming more than the extract supports |
| Cross-reference ≠ quote | A pointer to where a value lives is not the value being stated here |
| Derived number ≠ quoted | Computed/converted values are derived, never verbatim captures |
| Clause order must match | Re-ordering a quote silently changes its meaning |
| Cite a falsifiable defect | When two reviews disagree, the specific-location claim wins over generic approval |
| Verify until PASS, not for N rounds | A fix can address the cited example but not the invariant; the round that catches it is the loop working |