674 PDF Advanced Features 88df9439

PDF Advanced Features Guide

Comprehensive guide to advanced PDF extraction features (Priority 2 & 3).

Overview

Skill Seeker's PDF extractor now includes powerful advanced features for handling complex PDF scenarios:

Priority 2 Features (More PDF Types):

• ✅ OCR support for scanned PDFs
• ✅ Password-protected PDF support
• ✅ Complex table extraction

Priority 3 Features (Performance Optimizations):

• ✅ Parallel page processing
• ✅ Intelligent caching of expensive operations

Table of Contents

1. OCR Support for Scanned PDFs
2. Password-Protected PDFs
3. Table Extraction
4. Parallel Processing
5. Caching
6. Combined Usage
7. Performance Benchmarks

OCR Support

Extract text from scanned PDFs using Optical Character Recognition.

Installation

# Install Tesseract OCR engine
# Ubuntu/Debian
sudo apt-get install tesseract-ocr

# macOS
brew install tesseract

# Install Python packages
pip install pytesseract Pillow

Usage

# Basic OCR
python3 cli/pdf_extractor_poc.py scanned.pdf --ocr

# OCR with other options
python3 cli/pdf_extractor_poc.py scanned.pdf --ocr --verbose -o output.json

# Full skill creation with OCR
python3 cli/pdf_scraper.py --pdf scanned.pdf --name myskill --ocr

How It Works

1. Detection: For each page, checks if text content is < 50 characters
2. Fallback: If low text detected and OCR enabled, renders page as image
3. Processing: Runs Tesseract OCR on the image
4. Selection: Uses OCR text if it's longer than extracted text
5. Logging: Shows OCR extraction results in verbose mode

Example Output

📄 Extracting from: scanned.pdf
   Pages: 50
   OCR: ✅ enabled

  Page 1: 245 chars, 0 code blocks, 2 headings, 0 images, 0 tables
   OCR extracted 245 chars (was 12)
  Page 2: 389 chars, 1 code blocks, 3 headings, 0 images, 0 tables
   OCR extracted 389 chars (was 5)

Limitations

• Requires Tesseract installed on system
• Slower than regular text extraction (~2-5 seconds per page)
• Quality depends on PDF scan quality
• Works best with high-resolution scans

Best Practices

• Use --parallel with OCR for faster processing
• Combine with --verbose to see OCR progress
• Test on a few pages first before processing large documents

Password-Protected PDFs

Handle encrypted PDFs with password protection.

Usage

# Basic usage
python3 cli/pdf_extractor_poc.py encrypted.pdf --password mypassword

# With full workflow
python3 cli/pdf_scraper.py --pdf encrypted.pdf --name myskill --password mypassword

How It Works

1. Detection: Checks if PDF is encrypted (doc.is_encrypted)
2. Authentication: Attempts to authenticate with provided password
3. Validation: Returns error if password is incorrect or missing
4. Processing: Continues normal extraction if authentication succeeds

Example Output

📄 Extracting from: encrypted.pdf
   🔐 PDF is encrypted, trying password...
   ✅ Password accepted
   Pages: 100
   Metadata: {...}

Error Handling

# Missing password
❌ PDF is encrypted but no password provided
   Use --password option to provide password

# Wrong password
❌ Invalid password

Security Notes

• Password is passed via command line (visible in process list)
• For sensitive documents, consider environment variables
• Password is not stored in output JSON

Table Extraction

Extract tables from PDFs and include them in skill references.

Usage

# Extract tables
python3 cli/pdf_extractor_poc.py data.pdf --extract-tables

# With other options
python3 cli/pdf_extractor_poc.py data.pdf --extract-tables --verbose -o output.json

# Full skill creation with tables
python3 cli/pdf_scraper.py --pdf data.pdf --name myskill --extract-tables

How It Works

1. Detection: Uses PyMuPDF's find_tables() method
2. Extraction: Extracts table data as 2D array (rows × columns)
3. Metadata: Captures bounding box, row count, column count
4. Integration: Tables included in page data and summary

Example Output

📄 Extracting from: data.pdf
   Table extraction: ✅ enabled

  Page 5: 892 chars, 2 code blocks, 4 headings, 0 images, 2 tables
   Found table 0: 10x4
   Found table 1: 15x6

✅ Extraction complete:
   Tables found: 25

Table Data Structure

{
  "tables": [
    {
      "table_index": 0,
      "rows": [
        ["Header 1", "Header 2", "Header 3"],
        ["Data 1", "Data 2", "Data 3"],
        ...
      ],
      "bbox": [x0, y0, x1, y1],
      "row_count": 10,
      "col_count": 4
    }
  ]
}

Integration with Skills

Tables are automatically included in reference files when building skills:

## Data Tables

### Table 1 (Page 5)
| Header 1 | Header 2 | Header 3 |
|----------|----------|----------|
| Data 1   | Data 2   | Data 3   |

Limitations

• Quality depends on PDF table structure
• Works best with well-formatted tables
• Complex merged cells may not extract correctly

Parallel Processing

Process pages in parallel for 3x faster extraction.

Usage

# Enable parallel processing (auto-detects CPU count)
python3 cli/pdf_extractor_poc.py large.pdf --parallel

# Specify worker count
python3 cli/pdf_extractor_poc.py large.pdf --parallel --workers 8

# With full workflow
python3 cli/pdf_scraper.py --pdf large.pdf --name myskill --parallel --workers 8

How It Works

1. Worker Pool: Creates ThreadPoolExecutor with N workers
2. Distribution: Distributes pages across workers
3. Extraction: Each worker processes pages independently
4. Collection: Results collected and merged
5. Threshold: Only activates for PDFs with > 5 pages

Example Output

📄 Extracting from: large.pdf
   Pages: 500
   Parallel processing: ✅ enabled (8 workers)

🚀 Extracting 500 pages in parallel (8 workers)...

✅ Extraction complete:
   Total characters: 1,250,000
   Code blocks found: 450

Performance

Best Practices

• Use --workers equal to CPU core count
• Combine with --no-cache for first-time processing
• Monitor system resources (RAM, CPU)
• Not recommended for very large images (memory intensive)

Limitations

• Requires concurrent.futures (Python 3.2+)
• Uses more memory (N workers × page size)
• May not be beneficial for PDFs with many large images

Caching

Intelligent caching of expensive operations for faster re-extraction.

Usage

# Caching enabled by default
python3 cli/pdf_extractor_poc.py input.pdf

# Disable caching
python3 cli/pdf_extractor_poc.py input.pdf --no-cache

How It Works

1. Cache Key: Each page cached by page number
2. Check: Before extraction, checks cache for page data
3. Store: After extraction, stores result in cache
4. Reuse: On re-run, returns cached data instantly

What Gets Cached

• Page text and markdown
• Code block detection results
• Language detection results
• Quality scores
• Image extraction results
• Table extraction results

Example Output

  Page 1: Using cached data
  Page 2: Using cached data
  Page 3: 892 chars, 2 code blocks, 4 headings, 0 images, 0 tables

Cache Lifetime

• In-memory only (cleared when process exits)
• Useful for:
  • Testing extraction parameters
  • Re-running with different filters
  • Development and debugging

When to Disable

• First-time extraction
• PDF file has changed
• Different extraction options
• Memory constraints

Combined Usage

Maximum Performance

Extract everything as fast as possible:

python3 cli/pdf_scraper.py \
  --pdf docs/manual.pdf \
  --name myskill \
  --extract-images \
  --extract-tables \
  --parallel \
  --workers 8 \
  --min-quality 5.0

Scanned PDF with Tables

python3 cli/pdf_scraper.py \
  --pdf docs/scanned.pdf \
  --name myskill \
  --ocr \
  --extract-tables \
  --parallel \
  --workers 4

Encrypted PDF with All Features

python3 cli/pdf_scraper.py \
  --pdf docs/encrypted.pdf \
  --name myskill \
  --password mypassword \
  --extract-images \
  --extract-tables \
  --parallel \
  --workers 8 \
  --verbose

Performance Benchmarks

Test Setup

• Hardware: 8-core CPU, 16GB RAM
• PDF: 500-page technical manual
• Content: Mixed text, code, images, tables

Results

Feature Overhead

Troubleshooting

OCR Issues

Problem: pytesseract not found

# Install pytesseract
pip install pytesseract

# Install Tesseract engine
sudo apt-get install tesseract-ocr  # Ubuntu
brew install tesseract               # macOS

Problem: Low OCR quality

• Use higher DPI PDFs
• Check scan quality
• Try different Tesseract language packs

Parallel Processing Issues

Problem: Out of memory errors

# Reduce worker count
python3 cli/pdf_extractor_poc.py large.pdf --parallel --workers 2

# Or disable parallel
python3 cli/pdf_extractor_poc.py large.pdf

Problem: Not faster than sequential

• Check CPU usage (may be I/O bound)
• Try with larger PDFs (> 50 pages)
• Monitor system resources

Table Extraction Issues

Problem: Tables not detected

• Check if tables are actual tables (not images)
• Try different PDF viewers to verify structure
• Use --verbose to see detection attempts

Problem: Malformed table data

• Complex merged cells may not extract correctly
• Try extracting specific pages only
• Manual post-processing may be needed

Best Practices

For Large PDFs (500+ pages)

1. Use parallel processing:

   python3 cli/pdf_scraper.py --pdf large.pdf --parallel --workers 8
   

2. Extract to JSON first, then build skill:

   python3 cli/pdf_extractor_poc.py large.pdf -o extracted.json --parallel
   python3 cli/pdf_scraper.py --from-json extracted.json --name myskill
   

3. Monitor system resources

For Scanned PDFs

1. Use OCR with parallel processing:

   python3 cli/pdf_scraper.py --pdf scanned.pdf --ocr --parallel --workers 4
   

2. Test on sample pages first

3. Use --verbose to monitor OCR performance

For Encrypted PDFs

1. Use environment variable for password:

   export PDF_PASSWORD="mypassword"
   python3 cli/pdf_scraper.py --pdf encrypted.pdf --password "$PDF_PASSWORD"
   

2. Clear history after use to remove password

For PDFs with Tables

1. Enable table extraction:

   python3 cli/pdf_scraper.py --pdf data.pdf --extract-tables
   

2. Check table quality in output JSON

3. Manual review recommended for critical data

API Reference

PDFExtractor Class

from pdf_extractor_poc import PDFExtractor

extractor = PDFExtractor(
    pdf_path="input.pdf",
    verbose=True,
    chunk_size=10,
    min_quality=5.0,
    extract_images=True,
    image_dir="images/",
    min_image_size=100,
    # Advanced features
    use_ocr=True,
    password="mypassword",
    extract_tables=True,
    parallel=True,
    max_workers=8,
    use_cache=True
)

result = extractor.extract_all()

Configuration Options

Summary

✅ 6 Advanced Features implemented (Priority 2 & 3) ✅ 3x Performance Boost with parallel processing ✅ OCR Support for scanned PDFs ✅ Password Protection support ✅ Table Extraction from complex PDFs ✅ Intelligent Caching for faster re-runs

The PDF extractor now handles virtually any PDF scenario with maximum performance!