LLM Wiki Source Extraction Coverage

llm-wiki source extraction coverage

Silent extraction failure is the highest-frequency wiki defect today. When pdftotext returns 0 chars on an image-PDF, when python-docx skips embedded objects, when openpyxl ignores hidden sheets — the resulting wiki page looks complete but isn't. This skill closes the gap with two measurements per extraction and an anchor format for every wiki claim back to the source.

The pattern: predict coverage, measure coverage, inventory the gap, anchor every claim.

When this skill applies

The two metrics

extraction_estimate (pre-extraction)

Predicted upper-bound of what we expect to recover, before running the extractor. Computed from cheap structural inspection of the source.

Frontmatter field on the resulting sources/<slug>.md page:

extraction_estimate: 0.80
extraction_estimate_rationale: |
  PDF has 100 pages: 80 text-based (`pdffonts` shows embedded fonts), 20
  image-only (no embedded fonts, pixel-density consistent with scan). OCR
  fallback could lift toward 0.95 but baseline text-only is 0.80.

Range: 0.0 to 1.0. A 0.0 estimate means "this source is unreadable by current tooling without manual transcription".

extraction_yield (post-extraction)

Actual measured fraction recovered, after running the extractor and inspecting output.

extraction_yield: 0.94
extraction_yield_method: pdftotext+OCR  # how it was measured
extraction_yield_lost: |
  Page 47 OCR garbled — table 4-2 numeric values unreadable.
  Page 73 figure caption truncated mid-sentence.
  Pages 91-95 dense math, KaTeX transcription deferred.

Range: 0.0 to 1.0. If yield < estimate, the lost-content inventory is required — a bullet list of what was lost.

Measurement protocol per format

See format-specific references for exact commands. Common shape:

1. Count addressable units (pages / paragraphs / cells / slides)
2. Run extractor; count units with usable output
3. yield = units_recovered / units_total
4. Spot-check 5–10 random units against the source visually
5. List units that failed the spot-check in extraction_yield_lost

Format-specific routing

Existing implementation references:

• feedback_pdf_ocr_fallback_chain codifies the pdftotext → PyMuPDF → tesseract chain
• productivity/ocr-and-documents is the existing OCR skill (this skill cites, does not duplicate)
• engineering/doc-extraction is the engineering-specific extraction skill (used for technical PDFs)
• data/document-index-pipeline is the upstream ingestion pipeline that calls this skill

Source-anchor traceability

Every claim on a compiled wiki page (concepts/, standards/, methodology/) that derives from an extracted source must cite a precise location in the source. This is the revisability contract: a future reviewer can locate the original passage and verify or revise.

Anchor formats by source type

Anchor placement in compiled pages

In a concepts/ or standards/ page, anchors go at the end of the sentence they support, in parentheses:

The DNV-OS-E301 safety factor for ULS mooring conditions is **1.5**
([[sources/dnv-os-e301-2023]]:p47:¶2), reduced from the 2018 edition's
1.67 ([[sources/dnv-os-e301-2018]]:p41:¶3).

Multiple-source claims chain anchors:

Empirical yield in deepwater mooring failures clusters around 14% of
nameplate MBL ([[sources/sintef-2019-mooring-survey]]:p12:Table-3;
[[sources/api-bulletin-2tl]]:p8:¶4).

Anti-patterns

• Citing [[sources/<slug>]] without a sub-anchor → reviewer can't locate
• Anchor pointing at a page that lacks the claim (cut-and-paste error)
• Anchor in a section the extraction yield report flagged as lost
• Using anchor format for one source type on another (e.g., :p47 on a DOCX)

Pre-extraction protocol

Run before copying the binary into wikis/<domain>/sources/.

# 1. Identify format
file <source-path>

# 2. Cheap structural inspection (format-specific):
# PDF
pdfinfo <source-path>                            # pages, encrypted, etc.
pdffonts <source-path> | head                    # text-based vs scanned
# DOCX
unzip -l <source-path> | head                    # embedded objects, images
# XLSX
unzip -l <source-path> | grep sheet              # sheet count

# 3. Compute extraction_estimate (see format references for exact heuristics)
# 4. Record estimate in the source page frontmatter BEFORE extraction

For binaries >10 MB, do not copy into the wiki — create a ref pointer per llm-wiki-page-shape-contract Rule 3:

---
title: refs/<slug>
type: ref
external_path: /mnt/ace/<repo>/data/<file>.pdf
size: ~140 MB
extraction_estimate: 0.80                  # set even on ref pages
extraction_yield: null                      # filled in after compiled pages cite this ref
---

Post-extraction protocol

After running the extractor:

1. Measure: count addressable units recovered vs total (see format references for exact commands).
2. Spot-check: 5–10 random samples against the source. Visual or programmatic comparison.
3. Inventory loss: list every unit that didn't extract cleanly with page/paragraph/cell anchor + one-line reason.
4. Decide: is the yield enough to proceed?
   • Yield ≥ 0.90 AND no critical content lost → proceed to compile
   • Yield 0.50–0.90 → proceed but file an audit per research/llm-wiki-audit-feedback-loop with the loss inventory, so future passes know what to revisit
   • Yield < 0.50 → defer ingest; the source is not extractable enough to be useful. Note this in wikis/<domain>/CLAUDE.md "Open research questions" with the path and the failed yield.
5. Write frontmatter: extraction_yield, extraction_yield_method, extraction_yield_lost go on the sources/<slug>.md page.

Frontmatter required on sources/<slug>.md

---
title: sources/<slug>
type: source                                       # input layer per page-shape Rule 7
source_format: pdf | docx | xlsx | pptx | html | image | text
source_url: https://...                            # if applicable
source_path: /path/to/local/copy.pdf               # if binary copied in
external_path: /mnt/ace/<repo>/<file>              # if ref pointer (>10 MB)
date: YYYY-MM-DD                                   # original publication
ingested: YYYY-MM-DD                               # when extracted into wiki

# Extraction coverage (this skill's required fields)
extraction_estimate: 0.80
extraction_estimate_rationale: |
  <one-paragraph reason — what's recoverable, what isn't, why>
extraction_yield: 0.94
extraction_yield_method: pdftotext+OCR
extraction_yield_lost: |
  - Page 47: OCR garbled, table 4-2 numerics unreadable
  - Page 73: figure caption truncated mid-sentence
  - Pages 91–95: dense KaTeX, transcription deferred

# Wiki-shape contract fields
sources: []                                        # this IS a source; empty for source pages
tags: [<tag>]
license: <license-shorthand>
---

Decision tree per source

new source arrives
  │
  ├── plain text / Markdown ─────► yield = 1.0, no anchors needed beyond ¶
  │
  ├── HTML (clean) ──────────────► trafilatura; anchor by heading
  │
  ├── HTML (messy) ──────────────► trafilatura → BeautifulSoup fallback
  │
  ├── PDF
  │     ├── text-based ──────────► pdftotext -layout; anchor :p<page>:¶
  │     ├── mixed ───────────────► pdftotext + PyMuPDF where pdftotext = 0 chars
  │     └── scanned ─────────────► PyMuPDF render 300 DPI → tesseract --psm 6
  │
  ├── DOCX ──────────────────────► python-docx; fallback pandoc; anchor by ¶ id
  │
  ├── XLSX ──────────────────────► openpyxl per visible cell; anchor :<sheet>!<cell>
  │
  ├── PPTX ──────────────────────► python-pptx; anchor :slide<N>
  │
  └── image ─────────────────────► tesseract --psm 6; anchor :OCR

At every leaf: compute estimate before, yield after, write the inventory if yield < estimate, record anchor format for downstream cites.

Anti-patterns

• Writing a compiled page from a source whose extraction_yield was never recorded — invisible failure surface
• Yield = 1.0 claimed without spot-checking — overclaim
• Yield < 0.50 ingested anyway — pollutes the corpus
• Citing extracted content from a page whose lost-content inventory flagged that exact page → use the audit-feedback-loop to revise
• Reusing one source's anchor format on a different format (:p47 on a DOCX makes no sense)
• Storing the binary in the wiki when it's >10 MB instead of using a ref pointer per llm-wiki-page-shape-contract Rule 3
• Re-extracting a source repeatedly without recording the attempted yield upgrades — wastes compute, loses learning

What this skill is NOT

• Not a replacement for productivity/ocr-and-documents — that skill owns the OCR tooling specifics; this skill calls into it
• Not a replacement for engineering/doc-extraction — that's the engineering-domain extraction skill; this skill is the wiki-side contract for recording extraction quality
• Not a full RAG-replacement extractor — the extractor is the tool; this skill is the measurement and anchor contract
• Not for sources that are already plain text or clean HTML — those don't need pre/post metrics

Related must-fire rules

• feedback_pdf_ocr_fallback_chain — pdftotext+PyMuPDF=0 chars → image-PDF; fall back PyMuPDF 300 DPI → tesseract --psm 6
• feedback_runtime_base64_blocks_binary_roundtrip — JS tool results blocked binary; download path or save_to_disk for binary capture
• feedback_naive_secret_scan_false_positive_cascade — extracted content can contain false-positive regex matches; trust the hardened pre-commit hook
• feedback_subagent_write_phantom — if a subagent runs the extractor, main session must verify sources/<slug>.md actually landed on disk
• feedback_silent_verdict_flip_defect_class — extracted standards pages need section+edition, not just code_id