Large Document Familiarization

Large Document Familiarization

When to Use

• User says "review this", "get familiar with this", or "help out with this" while attaching a file >1 MB (or a multi-file export bundled into one HTML/zip snapshot).
• The document is an artifact bundle: manifests, digests, registers, templates — not a single narrative.
• You need a working model fast without burning the whole context budget on raw bytes.

Do NOT use for: small files you can read directly, or interactive/human-review workflows (see human-review).

The user frequently hands over big documents ("review this", "get familiar with this") — SOUL.md, research packets, master compliance packets, exported repo snapshots. A 5.8 MB / 140K-line file must NOT be read sequentially. Strategy: map structure cheaply, extract plain text once, jump to the sections that matter, then deliver a digest the user can correct and build on.

Steps

1. Size check before anything else.

   wc -c <file> && wc -l <file>
   

   5 MB or >50K lines → full structure-first flow below. Small files can be read directly.

2. Extract the heading outline (HTML only — cheapest map of the whole document):

   python3 -c "
   import re, html
   raw = open('FILE', encoding='utf-8', errors='replace').read()
   raw = re.sub(r'<script[\s\S]*?</script>', ' ', raw, flags=re.I)
   raw = re.sub(r'<style[\s\S]*?</style>', ' ', raw, flags=re.I)
   heads = re.findall(r'<h([1-6])[^>]*>(.*?)</h\1>', raw, flags=re.I|re.S)
   clean = lambda s: html.unescape(re.sub(r'<[^>]+>', ' ', s)).strip()
   for lvl, h in heads:
       print(f'H{lvl}: ' + re.sub(r'\s+', ' ', clean(h))[:120])
   " 2>/dev/null | head -200
   

   For markdown, grep -n '^#' works instead. The outline reveals section boundaries and file manifests — read it fully before touching content.

3. Dump readable text once to /tmp for cheap repeated reads:

   python3 -c "
   import re, html
   raw = open('FILE', encoding='utf-8', errors='replace').read()
   raw = re.sub(r'<script[\s\S]*?</script>', ' ', raw, flags=re.I)
   raw = re.sub(r'<style[\s\S]*?</style>', ' ', raw, flags=re.I)
   raw = re.sub(r'<[^>]+>', '\n', raw)
   text = html.unescape(raw)
   text = re.sub(r'[ \t]+', ' ', text)
   text = re.sub(r'\n{3,}', '\n\n', text)
   open('/tmp/doc.txt','w').write(text)
   print(f'text words: {len(text.split())}')
   "
   

4. Grep section boundaries by H1 titles, then read_file the core sections with offset/limit (~300-line chunks):

   grep -n -E "^(README|Title|...)" /tmp/doc.txt | head -40
   

5. Skim, don't read: manifests, artifact digests, byte tables, and file listings are metadata — skip past them to real content. Read core narrative docs in depth; spot-check templates/CSVs/JSON registers for status fields (e.g. complete_verified: 0, NOT_READY).

6. Deliver a working-model summary: what the document is (revision, digest, source), the system it describes, its current status/limits, key governance story, and the open decisions. End by asking what the user wants to do with it — the digest is a foundation, not the deliverable.

Pitfalls

• read_file truncates around 100K chars per call and is useless on a 5.8MB raw HTML file — always extract text first.
• HTML regex extraction with 2>/dev/null hides the "smart approval" flag noise; expected.
• Don't read section-by-section sequentially when the document is an artifact bundle — the heading outline shows you which files are real docs vs. manifests. Reading the manifest table wastes context.
• A document with a strong control discipline (compliance packets) usually has a "do-not-claim" / status register — that is often the single highest-signal section; read it even if you skim the rest.
• After building a working model of a specific large packet the user will return to, file the digest as references/<packet-name>.md under this skill so future sessions load it instead of re-reading the source.
• Existing digests: references/onyx-aerial-mesh-packet.md (Onyx Aerial Mesh R4-DRAFT, 2026-08-12) — a working model of that 5.8MB packet: system summary, compliance status, governance story, key internal documents.