Document Parsers

Document Parsers

Purpose: Autonomously parse and extract content from multiple document formats (PDF, DOCX, HTML, Markdown) using industry-standard libraries and AI-powered parsing tools.

Activation Triggers:

• Building RAG (Retrieval-Augmented Generation) pipelines
• Extracting text, tables, or metadata from documents
• Processing large document collections
• Converting documents to structured formats
• Handling complex PDFs with tables and layouts
• OCR for scanned documents
• Chunking documents for vector embeddings
• Building document search systems

Key Resources:

• scripts/setup-llamaparse.sh - Install and configure LlamaParse (AI-powered parsing)
• scripts/setup-unstructured.sh - Install Unstructured.io library
• scripts/parse-pdf.py - Functional PDF parser with multiple backend options
• scripts/parse-docx.py - DOCX document parser
• scripts/parse-html.py - HTML to structured text parser
• templates/multi-format-parser.py - Universal document parser template
• templates/table-extraction.py - Specialized table extraction template
• examples/parse-research-paper.py - Research paper parsing with citations
• examples/parse-legal-document.py - Legal document parsing with sections

Parser Comparison & Selection Guide

1. LlamaParse (AI-Powered Premium)

Best For:

• Complex PDFs with tables, charts, and mixed layouts
• Scanned documents requiring OCR
• Documents where accuracy is critical
• Multi-column layouts and scientific papers
• Financial reports and invoices

Pros:

• AI-powered layout understanding
• Excellent table extraction accuracy
• Built-in OCR support
• Handles complex formatting
• Structured output (Markdown/JSON)

Cons:

• Requires API key (paid service)
• API rate limits
• Network dependency
• Slower than local parsers

Documentation: https://docs.cloud.llamaindex.ai/llamaparse

Setup:

./scripts/setup-llamaparse.sh

Usage Pattern:

from llama_parse import LlamaParse

parser = LlamaParse(
    api_key="llx-...",
    result_type="markdown",  # or "text"
    language="en",
    verbose=True
)

documents = parser.load_data("document.pdf")
for doc in documents:
    print(doc.text)

2. Unstructured.io (Local Processing)

Best For:

• Batch processing many documents
• Multiple format support (PDF, DOCX, HTML, PPTX, Images)
• Local processing without API dependencies
• Structured element extraction
• Production RAG pipelines

Pros:

• Open-source and free
• Multi-format support
• Runs locally (no API keys)
• Good table detection
• Element-based chunking

Cons:

• Requires system dependencies (poppler, tesseract)
• Complex installation
• Less accurate than LlamaParse for complex layouts

Documentation: https://unstructured-io.github.io/unstructured/

Setup:

./scripts/setup-unstructured.sh

Usage Pattern:

from unstructured.partition.auto import partition

elements = partition("document.pdf")
for element in elements:
    print(f"{element.category}: {element.text}")

3. PyPDF2 (Simple PDF Text Extraction)

Best For:

• Simple text-based PDFs
• Quick prototyping
• Metadata extraction
• PDF manipulation (merge, split)

Pros:

• Pure Python (no dependencies)
• Fast and lightweight
• Good for simple PDFs
• Active maintenance

Cons:

• Poor table extraction
• Struggles with complex layouts
• No OCR support
• Limited formatting preservation

Documentation: https://github.com/py-pdf/pypdf2

Setup:

pip install pypdf2

Usage Pattern:

from PyPDF2 import PdfReader

reader = PdfReader("document.pdf")
for page in reader.pages:
    print(page.extract_text())

4. PDFPlumber (Advanced PDF Analysis)

Best For:

• Table extraction from PDFs
• PDF with tabular data
• Financial statements and reports
• Coordinate-based extraction

Pros:

• Excellent table extraction
• Visual debugging tools
• Coordinate-level control
• Metadata and layout info

Cons:

• Slower than PyPDF2
• Requires pdfminer.six dependency
• No OCR support

Documentation: https://github.com/jsvine/pdfplumber

Setup:

pip install pdfplumber

Usage Pattern:

import pdfplumber

with pdfplumber.open("document.pdf") as pdf:
    for page in pdf.pages:
        tables = page.extract_tables()
        text = page.extract_text()

5. python-docx (Word Documents)

Best For:

• Microsoft Word (.docx) documents
• Extracting paragraphs, tables, headers
• Document metadata
• Template-based document generation

Pros:

• Native DOCX support
• Preserves structure (paragraphs, tables, sections)
• Access to styles and formatting
• Can also write/modify DOCX

Cons:

• Only works with .docx (not .doc)
• Limited image extraction

Documentation: https://github.com/python-openxml/python-docx

Setup:

pip install python-docx

Usage Pattern:

from docx import Document

doc = Document("document.docx")
for para in doc.paragraphs:
    print(para.text)
for table in doc.tables:
    for row in table.rows:
        print([cell.text for cell in row.cells])

Decision Matrix

Functional Scripts

1. Parse PDF (scripts/parse-pdf.py)

Command-line PDF parser supporting multiple backends:

# Using PyPDF2 (default)
python scripts/parse-pdf.py document.pdf

# Using PDFPlumber (better for tables)
python scripts/parse-pdf.py document.pdf --backend pdfplumber

# Using LlamaParse (AI-powered)
python scripts/parse-pdf.py document.pdf --backend llamaparse --api-key llx-...

# Output to file
python scripts/parse-pdf.py document.pdf --output output.txt

# Extract tables as JSON
python scripts/parse-pdf.py document.pdf --backend pdfplumber --tables-only --output tables.json

Features:

• Multiple backend support (PyPDF2, PDFPlumber, LlamaParse)
• Table extraction
• Metadata extraction
• Page range selection
• JSON/Text output formats

2. Parse DOCX (scripts/parse-docx.py)

Word document parser with structure preservation:

# Basic extraction
python scripts/parse-docx.py document.docx

# Extract with structure
python scripts/parse-docx.py document.docx --preserve-structure

# Extract tables only
python scripts/parse-docx.py document.docx --tables-only

# Output as JSON
python scripts/parse-docx.py document.docx --output output.json --format json

Features:

• Paragraph extraction with styles
• Table extraction
• Header/footer extraction
• Metadata (author, created date, etc.)
• Structured JSON output

3. Parse HTML (scripts/parse-html.py)

HTML to clean text converter:

# Basic HTML parsing
python scripts/parse-html.py document.html

# From URL
python scripts/parse-html.py https://example.com/article

# Preserve links
python scripts/parse-html.py document.html --preserve-links

# Extract specific selector
python scripts/parse-html.py document.html --selector "article.content"

Features:

• Clean text extraction (removes scripts, styles)
• Link preservation
• CSS selector support
• URL fetching
• Markdown output option

Templates

Multi-Format Parser (templates/multi-format-parser.py)

Universal parser handling multiple formats with automatic format detection:

from multi_format_parser import MultiFormatParser

parser = MultiFormatParser(
    llamaparse_api_key="llx-...",  # Optional
    use_ocr=True,
    chunk_size=1000
)

# Automatic format detection
result = parser.parse_file("document.pdf")
print(result.text)
print(result.metadata)
print(result.tables)

# Batch processing
results = parser.parse_directory("./documents/")
for filename, result in results.items():
    print(f"{filename}: {len(result.text)} characters")

Supports:

• PDF, DOCX, HTML, Markdown, TXT
• Automatic chunking for RAG
• Metadata extraction
• Table extraction across all formats
• Error handling and fallbacks

Table Extraction (templates/table-extraction.py)

Specialized table extraction with multiple strategies:

from table_extraction import TableExtractor

extractor = TableExtractor(
    prefer_llamaparse=True,
    fallback_to_pdfplumber=True
)

# Extract all tables from document
tables = extractor.extract_tables("financial_report.pdf")

for i, table in enumerate(tables):
    print(f"Table {i + 1}:")
    print(table.to_markdown())  # or .to_csv(), .to_json()
    print(f"Confidence: {table.confidence}")

Features:

• Multiple extraction strategies
• Automatic fallback
• Table validation
• Format conversion (CSV, JSON, Markdown, DataFrame)
• Confidence scoring

Examples

Research Paper Parsing (examples/parse-research-paper.py)

Complete example for parsing academic papers:

# Extracts title, abstract, sections, citations, tables, figures
python examples/parse-research-paper.py paper.pdf --output paper.json

Extracts:

• Title and authors
• Abstract
• Section structure (Introduction, Methods, Results, etc.)
• Citations and references
• Tables and figures with captions
• Metadata (DOI, publication date, journal)

Legal Document Parsing (examples/parse-legal-document.py)

Specialized parser for legal documents:

# Extracts clauses, sections, definitions, parties
python examples/parse-legal-document.py contract.pdf --output contract.json

Extracts:

• Document type (contract, agreement, etc.)
• Parties involved
• Definitions section
• Numbered clauses and sections
• Signature blocks
• Dates and deadlines

RAG Pipeline Integration

Document Chunking for Embeddings

from multi_format_parser import MultiFormatParser

parser = MultiFormatParser(chunk_size=512, chunk_overlap=50)
result = parser.parse_file("document.pdf")

# Chunks ready for embedding
for chunk in result.chunks:
    print(f"Chunk {chunk.id}: {chunk.text[:100]}...")
    print(f"Metadata: {chunk.metadata}")
    # Send to embedding model

Batch Processing Pipeline

import glob
from multi_format_parser import MultiFormatParser

parser = MultiFormatParser()

# Process all documents in directory
for filepath in glob.glob("./documents/**/*", recursive=True):
    try:
        result = parser.parse_file(filepath)
        # Store in vector database
        store_embeddings(result.chunks)
        print(f"✓ Processed {filepath}")
    except Exception as e:
        print(f"✗ Failed {filepath}: {e}")

Best Practices

Parser Selection:

• Start with PyPDF2 for simple PDFs, upgrade if needed
• Use LlamaParse for complex layouts (budget permitting)
• Use Unstructured for multi-format production systems
• Use PDFPlumber specifically for table extraction

Performance:

• Cache parsed results to avoid re-processing
• Use batch processing for multiple documents
• Consider async processing for large collections
• Monitor API rate limits for LlamaParse

Accuracy:

• Validate table extraction results
• Implement fallback strategies
• Log parsing errors for debugging
• Use confidence scores when available

RAG Optimization:

• Chunk size: 512-1024 tokens for embeddings
• Overlap: 10-20% for context preservation
• Preserve metadata (page numbers, sections) for retrieval
• Clean extracted text (remove headers/footers)

Troubleshooting

PyPDF2 returns garbled text:

• Try PDFPlumber or LlamaParse
• PDF may have non-standard encoding
• Check if PDF is scanned (needs OCR)

Unstructured installation fails:

• Install system dependencies: sudo apt-get install poppler-utils tesseract-ocr
• On macOS: brew install poppler tesseract

LlamaParse API errors:

• Verify API key is correct
• Check rate limits in dashboard
• Ensure document size is within limits

Table extraction misses columns:

• Try different parser (PDFPlumber vs LlamaParse)
• Adjust table detection settings
• Validate table structure manually

DOCX parsing fails:

• Ensure file is .docx not .doc
• Check file is not corrupted
• Try converting to .docx with LibreOffice

Dependencies

Core:

pip install pypdf2 pdfplumber python-docx beautifulsoup4 lxml markdown

Optional (Unstructured):

pip install unstructured[local-inference]
sudo apt-get install poppler-utils tesseract-ocr  # Linux
brew install poppler tesseract  # macOS

Optional (LlamaParse):

pip install llama-parse
# Requires API key from https://cloud.llamaindex.ai

Supported Formats: PDF, DOCX, HTML, Markdown, TXT Parsers: LlamaParse, Unstructured.io, PyPDF2, PDFPlumber, python-docx Version: 1.0.0