# Intel Bytes

> Identify every basin-attributed numeric value in a US/Canada onshore oil & gas report and enter it as a "byte" into the 39-column Skeleton XLSX.

- Skill: `darvin2802/intel-bytes` (Agent Skill, multi-file: 17 files)
- Install (CLI): `npx skillmds@latest add darvin2802/intel-bytes`
- Raw SKILL.md: https://api.skillmd.com/api/skills/darvin2802/intel-bytes/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Data & Analytics
- Author: Darvin2802 (https://skillmd.com/u/darvin2802)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/darvin2802/intel-bytes

---


# Skill-Driven Extraction - Identification & Entry (v0.7)

## STOP — required reading gate (blocking, EVERY run)

**No extraction begins until every box below is confirmed loaded into context.** Every PDF is processed the same way — there is no "cold" vs "production" mode. This gate exists because on 2026-07-01 a run regressed 78.5% vs a prior 97.8% on the same PDF — root cause was skipping `rules/rules.md`. See [the 20pp regression lesson](/docs/engineering-notes/regression-lesson.md).

Required-reading checklist for EVERY run (no exceptions):
- [ ] `Reference/Bytes_Entry_Process_Document.md` — the SOP (WHAT to extract)
- [ ] `skill/SKILL.md` — this file, the pipeline (HOW to run)
- [ ] `skill/rules/rules.md` — enforced rule registry (the WHICH rules)
- [ ] `memory/MEMORY.md` — session doctrine index

Freshness check BEFORE loading rules.md:
```
python3 skill/scripts/check_rules_fresh.py
```
Fails loudly if `rules.md` is older than `rules.yaml` (generator was skipped). Regenerate with `python3 rules/gen_rules_md.py` and retry.

The very first task in every task list must be "Load and confirm required-reading checklist." Do not create extraction/populate/audit tasks until the gate passes. If a gold XLSX is later uploaded, the diff is a follow-up action (not a separate mode) — see [[feedback-extraction-protocol]].

## Pipeline overview

This skill is a **5-stage pipeline**. All extraction/normalization RULES live in one place:
**`rules/rules.md`** (generated from `rules/rules.yaml`) - the single source of truth. This file
NEVER restates a rule; it tells you WHICH rules apply at each stage and references them by id
(e.g. `subloc.cascade`). Confidence governs enforcement: **locked** = auto-fixed by the validator,
**confirmed** = flagged, **candidate/quarantine** = do NOT apply.

Read once per session (not per PDF): this file, `rules/rules.md`, and the references you need.

## 0 - Inputs
- New report PDF in `inputs/`.
- `reference/` SOPs + lookups; `Reference/US Skeleton 3.xlsx` (39-col template); `2026 Tracker.xlsx` (company -> Peer Group).
- Deep how-to detail in `references/01..07` (field guide, ARPD construction, parameter reference, lookups, edge cases, output schema, validation checklist). These explain HOW; `rules/` defines WHAT.

## Pipeline

### 1. EXTRACT  (deterministic)
- Text via `pdftotext -layout` (poppler); pdfplumber only as image-only fallback. ≥25-page PDFs: delegate extraction to a subagent to avoid context bloat.

### 2. IDENTIFY  (capture-by-default)
- Apply `scope.*` rules: capture EVERY basin-attributed number unless it matches the closed skip-list (`scope.skiplist`). Read every page/section in full (`scope.page_scan`). Capture mix % (`scope.mix_pct`) and every instance (`scope.every_instance`).
- Output: raw candidate bytes (value + source location + the report's own context).

### 3. NORMALIZE  (apply the rule registry, by column)
For each candidate byte, set each column per its rule namespace in `rules/rules.md`:
- Sub-Location -> `subloc.*` (full cascade)
- Currency -> `currency.*`     Unit -> `unit.*`     Period -> `period.*`     Actual/Forecast -> `af.*`
- Parameter -> `param.*`, `reserves.*`, `pv10.*`, `price.*`     Play/Formation -> `play.*`
- Parameter detail / DPD -> `pd.*`     ARPD -> `arpd.*` + `references/02_arpd_construction.md`
- Deterministic transforms run via `scripts/` (see each rule's `enforced_by`); judgment rules you apply directly.

### 4. VALIDATE  (re-check the same rules)
- `python3 scripts/validate_output.py --xlsx <out>.xlsx --report-type "<type>" --pdf-pages N --country <USA|Canada> --auto-fix`
    (enforces the locked/confirmed rules; auto-fixes or flags.)
- `python3 scripts/audit_coverage.py --pdf <in>.pdf --xlsx <out>.xlsx` - the miss-catcher gate. **Resolve EVERY flag individually** (capture it, or give a specific per-item skip reason). Bulk-dismissal is forbidden (the #19 28-miss cause).

### 5. EMIT
- Write the 39-col skeleton (`references/06_output_schema.md`) to `outputs/` named after the input PDF. Add a Flags sheet for anomalies. Apply `emit.*` rules (sheet-name artifact, source-title).
- Highlighted PDF: **NEVER generated** — permanent 2026-07-01 decision per [[feedback-pause-highlighting-during-dev]]. Do not invoke any highlighting pipeline in any scenario.

## Output schema
39 columns - see `references/06_output_schema.md`. Sheet name = report name truncated to 31 chars (`emit.sheet_name_artifact`).

## Maintaining rules (so we stop the diff-accretion)
- Edit ONLY `rules/rules.yaml`, then regenerate: `python3 rules/gen_rules_md.py`. Never hand-edit `rules.md`.
- Before/after any rule or script change: `python3 tests/regression.py` (must stay green) and `python3 tests/score.py --mine <out> --gold <gold>` on the corpus.
- New rule lifecycle: candidate (1 run) -> confirmed (2) -> locked (3+). Only `locked` rules are auto-fixed.
- The long files truncate when edited with the file tool - edit scripts/SKILL via bash, recompile/verify after.