PDF Extract

PDF Extract

Extract structured data from PDFs using schema-driven field extraction. A generalized, extensible framework that works with ANY document type - property assessments, utility bills, inspection reports, financial statements, etc.

No MCP dependencies - Uses local Python libraries only (pdfplumber, PyYAML)

Storage: Schemas in skills/pdf-extract/schemas/, outputs to JSON

What This Skill Does

1. Schema-Driven Extraction: Define what to extract using YAML schemas
2. Multiple Strategies: Table cell matching, text proximity, regex fallback
3. Batch Processing: Extract from multiple PDFs with glob patterns
4. Exploration Mode: Dump raw PDF content to design schemas

When to Use This Skill

Use pdf-extract when:

• Need to extract structured fields from PDFs (property data, utility usage, inspection findings)
• Have multiple similar documents to process (batch extraction)
• Want to convert unstructured PDFs to JSON for RAG indexing
• User asks to "extract X from these PDFs" or "parse property reports"

Don't use when:

• PDF is just text to read (use Read tool directly)
• Need full semantic search (use Soapbox RAG (rag_ingest MCP tool) instead)
• Document is already structured (CSV, JSON, etc.)

Setup: Install Dependencies

Before first use, install required Python packages:

pip install --break-system-packages pdfplumber PyYAML pytest

Workflow Option 1: Automatic Schema Discovery (Recommended)

When you have 3+ similar PDFs, use automatic schema discovery:

# Automatically discover schemas and extract data
python3 skills/pdf-extract/scripts/auto_extract.py documents/*.pdf --output results/

What happens:

1. Groups similar PDFs by page count and structure
2. Discovers common fields across each group (3+ documents)
3. Generates schemas automatically
4. Extracts data using discovered schemas
5. Saves schemas and extracted data

Output:

results/
├── schema_6-15_pages.yaml         # Auto-discovered schema
├── extracted_6-15_pages.json       # Extracted data
└── all_extractions.json            # Combined results

Or discover schema only (without extraction):

# Discover and save schema
python3 skills/pdf-extract/scripts/discover_schemas.py documents/*.pdf \
  --output schemas/my_schema.yaml \
  --min-confidence 0.7

Then review and refine the schema before extraction.

Workflow Option 2: Manual Schema Creation

Step 1: Explore PDF Content (Design Schema)

For custom schemas or when you have fewer than 3 documents, explore the PDF to see how data is structured:

python3 skills/pdf-extract/scripts/pdf_extract.py document.pdf \
  --dump-text --dump-tables --pages 1-6

What to look for:

• How labels appear ("Property name:", "Year Built", etc.)
• Whether data is in tables or plain text
• Page numbers where key information appears

Step 2: Create Extraction Schema

Create a YAML schema defining what fields to extract. Use the example template:

# Copy template as starting point
cp skills/pdf-extract/schemas/example_document.yaml \
   skills/pdf-extract/schemas/my_document.yaml

# Edit to match your document structure

Schema Format:

fields:
  property_name:
    label: Property Name              # Human-readable name
    type: text                        # text | number | date
    search_terms:                     # Labels to search for
      - "Property name"
      - "Subject Property"
    fallback:                         # Alternative terms (optional)
      - "Project name"
    page_range: [1, 3]                # Pages to search (optional, 1-indexed)
    pattern: "regex pattern"          # Validation/extraction regex (optional)

Field Types:

• text - String values (names, addresses, IDs)
• number - Integers or floats (commas removed automatically)
• date - Date values (future enhancement)

Extraction Strategies: The framework tries multiple strategies in order:

1. Table cell extraction (highest confidence) - finds values in table cells adjacent to labels
2. Text proximity (high confidence) - finds values near labels (same line or next line)
3. Regex fallback (medium/low confidence) - searches with patterns when other methods fail

Step 3: Extract from Single PDF

Extract structured data from a single PDF:

python3 skills/pdf-extract/scripts/pdf_extract.py document.pdf \
  --schema skills/pdf-extract/schemas/my_document.yaml

Output (JSON to stdout):

[
  {
    "source_file": "document.pdf",
    "fields": {
      "property_name": {
        "value": "Carolina Club",
        "page": 1,
        "confidence": "high",
        "method": "text_proximity"
      },
      "year_built": {
        "value": 2002,
        "page": 3,
        "confidence": "high",
        "method": "table_cell"
      }
    }
  }
]

Confidence Levels:

• high - Found in table cell or same line as label
• medium - Found on line after label, or via regex near label
• low - Found via regex with no nearby label
• none - Not found

Step 4: Batch Extraction

Extract from multiple PDFs using glob patterns:

python3 skills/pdf-extract/scripts/pdf_extract.py "documents/*.pdf" \
  --schema skills/pdf-extract/schemas/my_document.yaml \
  --output extracted_data.json

Progress output to stderr:

✓ Extracted from documents/building_a.pdf
✓ Extracted from documents/building_b.pdf
✗ Failed to extract from documents/corrupted.pdf: Invalid PDF
Results written to extracted_data.json

Integration with Local RAG

The PDF extraction framework works great with the rag_ingest skill:

Workflow:

1. Extract structured data from PDFs → JSON
2. Index extracted fields into RAG for semantic search
3. Query to find buildings/properties by any field

Example:

# 1. Extract data from property assessments
python3 skills/pdf-extract/scripts/pdf_extract.py "assessments/*.pdf" \
  --schema skills/pdf-extract/schemas/property_assessment.yaml \
  --output property_data.json

# 2. Index the extracted JSON into RAG
# (Future enhancement: auto-index extracted data)

# 3. Query: "Find all buildings built before 2000"
python3 skills/rag_ingest/scripts/query.py \
  --query "buildings constructed before year 2000" \
  --top-k 5

Creating Custom Schemas

Example: Utility Bill Extraction

# schemas/utility_bill.yaml
fields:
  account_number:
    label: Account Number
    type: text
    search_terms: ["Account #", "Account No", "Acct"]
    page_range: [1, 1]
    pattern: "\\d{10,12}"
    
  billing_period:
    label: Billing Period
    type: text
    search_terms: ["Billing Period", "Service Period"]
    page_range: [1, 1]
    
  total_kwh:
    label: Total kWh Usage
    type: number
    search_terms: ["Total kWh", "Total Usage", "Energy Used"]
    page_range: [1, 2]
    pattern: "[\\d,]+"
    
  total_cost:
    label: Total Amount Due
    type: number
    search_terms: ["Amount Due", "Total Due", "Balance"]
    page_range: [1, 2]
    pattern: "\\$[\\d,]+\\.\\d{2}"

Example: Inspection Report Extraction

# schemas/inspection_report.yaml
fields:
  inspection_date:
    label: Inspection Date
    type: date
    search_terms: ["Date of Inspection", "Inspection Date"]
    page_range: [1, 1]
    pattern: "\\d{1,2}/\\d{1,2}/\\d{4}"
    
  inspector_name:
    label: Inspector Name
    type: text
    search_terms: ["Inspector", "Inspected by"]
    page_range: [1, 1]
    
  building_condition:
    label: Overall Condition
    type: text
    search_terms: ["Overall Condition", "General Condition", "Rating"]
    page_range: [1, 3]
    
  critical_findings:
    label: Critical Findings Count
    type: number
    search_terms: ["Critical Findings", "Immediate Action Required"]
    page_range: [1, 10]

Troubleshooting

Field not found:

• Check page_range includes the right pages (use --dump-text to verify)
• Add more search_terms or fallback terms
• Use --dump-text to see raw text and verify label exists

Wrong value extracted:

• Add a pattern to validate/filter the extracted value
• Check if data is in a table (use --dump-tables)
• Tighten page_range to avoid wrong section

Low confidence:

• Data in tables = highest confidence
• Data next to label = high confidence
• Data found by regex confidence
• Consider restructuring schema to improve extraction

Architecture

pdf_extract.py          # Main CLI and orchestration
├── schema_loader.py    # Load and validate YAML/JSON schemas
├── pdf_processor.py    # Extract text and tables with pdfplumber
├── field_extractor.py  # Orchestrate extraction strategies
└── extractors.py       # Individual extraction strategies
    ├── TextProximityExtractor
    ├── TableCellExtractor
    └── RegexExtractor

Testing

Run the test suite to verify everything works:

cd skills/pdf-extract/scripts
pytest -v

Expected output:

19 passed in 0.30s

Advanced Usage

Page Range Optimization

For large PDFs, limit extraction to relevant pages:

fields:
  executive_summary:
    label: Executive Summary
    type: text
    search_terms: ["Executive Summary", "Summary"]
    page_range: [1, 3]  # Only search first 3 pages
    
  detailed_findings:
    label: Detailed Findings
    type: text
    search_terms: ["Findings", "Analysis"]
    page_range: [10, 50]  # Skip first pages

Pattern Validation

Use regex patterns to validate extracted values:

fields:
  year_built:
    label: Year Built
    type: number
    search_terms: ["Year Built", "Construction Year"]
    pattern: "(19|20)\\d{2}"  # Only 1900-2099
    
  zip_code:
    label: ZIP Code
    type: text
    search_terms: ["ZIP", "Zip Code"]
    pattern: "\\d{5}(-\\d{4})?"  # ZIP or ZIP+4

Example: Full Workflow

# 1. Explore a sample PDF
python3 skills/pdf-extract/scripts/pdf_extract.py sample_report.pdf \
  --dump-text --pages 1-3

# 2. Create schema based on exploration
# Edit: skills/pdf-extract/schemas/my_schema.yaml

# 3. Test on single PDF
python3 skills/pdf-extract/scripts/pdf_extract.py sample_report.pdf \
  --schema skills/pdf-extract/schemas/my_schema.yaml

# 4. Batch process all PDFs
python3 skills/pdf-extract/scripts/pdf_extract.py "reports/*.pdf" \
  --schema skills/pdf-extract/schemas/my_schema.yaml \
  --output all_reports.json

# 5. Use extracted data
cat all_reports.json | jq '.[] | {name: .fields.property_name.value, year: .fields.year_built.value}'