Intel Bytes

Skill-Driven Extraction - Identification & Entry (v0.7)

STOP — required reading gate (blocking, EVERY run)

No extraction begins until every box below is confirmed loaded into context. Every PDF is processed the same way — there is no "cold" vs "production" mode. This gate exists because on 2026-07-01 a run regressed 78.5% vs a prior 97.8% on the same PDF — root cause was skipping rules/rules.md. See the 20pp regression lesson.

Required-reading checklist for EVERY run (no exceptions):

• Reference/Bytes_Entry_Process_Document.md — the SOP (WHAT to extract)
• skill/SKILL.md — this file, the pipeline (HOW to run)
• skill/rules/rules.md — enforced rule registry (the WHICH rules)
• memory/MEMORY.md — session doctrine index

Freshness check BEFORE loading rules.md:

python3 skill/scripts/check_rules_fresh.py

Fails loudly if rules.md is older than rules.yaml (generator was skipped). Regenerate with python3 rules/gen_rules_md.py and retry.

The very first task in every task list must be "Load and confirm required-reading checklist." Do not create extraction/populate/audit tasks until the gate passes. If a gold XLSX is later uploaded, the diff is a follow-up action (not a separate mode) — see [[feedback-extraction-protocol]].

Pipeline overview

This skill is a 5-stage pipeline. All extraction/normalization RULES live in one place: rules/rules.md (generated from rules/rules.yaml) - the single source of truth. This file NEVER restates a rule; it tells you WHICH rules apply at each stage and references them by id (e.g. subloc.cascade). Confidence governs enforcement: locked = auto-fixed by the validator, confirmed = flagged, candidate/quarantine = do NOT apply.

Read once per session (not per PDF): this file, rules/rules.md, and the references you need.

0 - Inputs

• New report PDF in inputs/.
• reference/ SOPs + lookups; Reference/US Skeleton 3.xlsx (39-col template); 2026 Tracker.xlsx (company -> Peer Group).
• Deep how-to detail in references/01..07 (field guide, ARPD construction, parameter reference, lookups, edge cases, output schema, validation checklist). These explain HOW; rules/ defines WHAT.

Pipeline

1. EXTRACT (deterministic)

• Text via pdftotext -layout (poppler); pdfplumber only as image-only fallback. ≥25-page PDFs: delegate extraction to a subagent to avoid context bloat.

2. IDENTIFY (capture-by-default)

• Apply scope.* rules: capture EVERY basin-attributed number unless it matches the closed skip-list (scope.skiplist). Read every page/section in full (scope.page_scan). Capture mix % (scope.mix_pct) and every instance (scope.every_instance).
• Output: raw candidate bytes (value + source location + the report's own context).

3. NORMALIZE (apply the rule registry, by column)

For each candidate byte, set each column per its rule namespace in rules/rules.md:

• Sub-Location -> subloc.* (full cascade)
• Currency -> currency.* Unit -> unit.* Period -> period.* Actual/Forecast -> af.*
• Parameter -> param.*, reserves.*, pv10.*, price.* Play/Formation -> play.*
• Parameter detail / DPD -> pd.* ARPD -> arpd.* + references/02_arpd_construction.md
• Deterministic transforms run via scripts/ (see each rule's enforced_by); judgment rules you apply directly.

4. VALIDATE (re-check the same rules)

• python3 scripts/validate_output.py --xlsx <out>.xlsx --report-type "<type>" --pdf-pages N --country <USA|Canada> --auto-fix (enforces the locked/confirmed rules; auto-fixes or flags.)
• python3 scripts/audit_coverage.py --pdf <in>.pdf --xlsx <out>.xlsx - the miss-catcher gate. Resolve EVERY flag individually (capture it, or give a specific per-item skip reason). Bulk-dismissal is forbidden (the #19 28-miss cause).

5. EMIT

• Write the 39-col skeleton (references/06_output_schema.md) to outputs/ named after the input PDF. Add a Flags sheet for anomalies. Apply emit.* rules (sheet-name artifact, source-title).
• Highlighted PDF: NEVER generated — permanent 2026-07-01 decision per [[feedback-pause-highlighting-during-dev]]. Do not invoke any highlighting pipeline in any scenario.

Output schema

39 columns - see references/06_output_schema.md. Sheet name = report name truncated to 31 chars (emit.sheet_name_artifact).

Maintaining rules (so we stop the diff-accretion)

• Edit ONLY rules/rules.yaml, then regenerate: python3 rules/gen_rules_md.py. Never hand-edit rules.md.
• Before/after any rule or script change: python3 tests/regression.py (must stay green) and python3 tests/score.py --mine <out> --gold <gold> on the corpus.
• New rule lifecycle: candidate (1 run) -> confirmed (2) -> locked (3+). Only locked rules are auto-fixed.
• The long files truncate when edited with the file tool - edit scripts/SKILL via bash, recompile/verify after.