# Source Extraction Coverage

> Runs doc-type-aware deterministic extraction on a source and makes shallow extraction visible by recording an extraction_estimate before and an extraction_yield after, flagging any source whose yield falls short. Use when ingesting a new source file (PDF/DOCX/XLSX/HTML/scanned) into the provisional store.

- Skill: `vamseeachanta/source-extraction-coverage` (Agent Skill, multi-file: 2 files)
- Install (CLI): `npx skillmds@latest add vamseeachanta/source-extraction-coverage`
- Raw SKILL.md: https://api.skillmd.com/api/skills/vamseeachanta/source-extraction-coverage/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Web & Frontend
- License: CC-BY-4.0
- Author: vamseeachanta (https://skillmd.com/u/vamseeachanta)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/vamseeachanta/source-extraction-coverage

---


# source-extraction-coverage

> Template skill (doc 08, doc 01). The estimate/yield pair is the whole point:
> it is how the 2%-coverage failure (LLM-extracted a doc and silently produced
> ~2% of its text) becomes a loud, machine-checkable signal instead of a silent
> loss.

## Trigger
`/extract-source <path> [--type auto|pdf|docx|xlsx|html|scanned]`

## Preconditions
1. Source lives in the off-repo read-only archive; record its `sha256`.
2. Type is detected by content/structure, **not filename** (incident D4:
   names lie about content).

## Steps
1. **Estimate first.** Before extracting, declare `extraction_estimate` in the
   landing-page frontmatter: roughly how much extractable content this source
   holds (page count, table count, "text-heavy" vs "image-only"). This is a
   commitment made *before* you see the result, so a shortfall is undeniable.
2. **Pick the recipe by type** (use ADOPT-tier permissive tools from doc 12):
   - `pdf` (born-digital): deterministic text+geometry (pdfplumber/Docling);
     `pdf` (scanned/image-only) → route to the `scanned` recipe (OCR as
     labeled interpretation, doc 11).
   - `docx`: structured access (python-docx) + semantic Markdown (mammoth).
   - `xlsx`: formula-graph read (openpyxl) / fast data read (calamine) —
     classify by scanned structure, not sheet names (D4).
   - `html`: typed-element extraction; strip nav/boilerplate.
   - `scanned`: OCR with a blur gate (OpenCV Laplacian-variance → too blurry
     stays provisional); OCR output is a *claim*, not source text.
   - `text/markup`: section-aware deterministic copy with paragraph or section
     estimate/yield.
   - `small-json/config`: key/schema-aware metadata extraction with generated
     JSON excluded before extraction.
   - `code-doc`: docstring/comment metadata only; never recursive source-tree
     ingestion.
3. **Extract deterministically.** Tools, not an LLM, do the bulk copy. (LLM
   bulk extraction yielded ~2% coverage — never the bulk path.)
4. **Record yield.** Write `extraction_yield` after extraction (pages/tables
   actually produced, % of estimate).
5. **Flag shortfall.** If `yield << estimate`, mark the source
   `extraction: shallow` and queue it for a different recipe. Do not let it
   pass as "done."

## Verification
- Frontmatter has BOTH `extraction_estimate` (pre) and `extraction_yield`
  (post); a validator script rejects a page missing either.
- Sources flagged `shallow` appear in the re-extraction backlog, not the
  trusted set.
- Generated/repetitive JSON and source-tree noise are exclusions, not shallow
  extraction failures.

## Cleanup
- Raw source never copied into any repo; only derived parts + a `sources:`
  sha256 pointer.

## Incident appendix
| Rule | Why |
|---|---|
| Estimate before, yield after | The 2%-coverage loss was invisible until the pair made it loud |
| Classify on structure not name | D4: sheet/file names misclassify content |
| OCR output is a claim | doc 11: scanned text is interpretation, enters the provisional gate |