Process Pdfs

PDF Processing Guide

Overview

This guide covers essential PDF processing operations using Python libraries and command-line tools. For advanced features, JavaScript libraries, and detailed examples, see reference.md. If you need to fill out a PDF form, read forms.md and follow its instructions.

Setup

Python Dependencies

Install required libraries:

# Core libraries
pip install pypdf pdfplumber reportlab

# For OCR (scanned PDFs)
pip install pytesseract pdf2image

# For data export
pip install pandas openpyxl

Command-Line Tools

macOS (using Homebrew):

brew install poppler qpdf tesseract

Linux (Debian/Ubuntu):

sudo apt-get install poppler-utils qpdf tesseract-ocr

Working Directory

When using scripts from the scripts/ directory, run them from the skill directory:

cd /Users/markayers/.claude/skills/process-pdfs
python scripts/script_name.py

Or use absolute paths when referencing scripts.

Quick Start

from pypdf import PdfReader, PdfWriter

# Read a PDF
reader = PdfReader("document.pdf")
print(f"Pages: {len(reader.pages)}")

# Extract text
text = ""
for page in reader.pages:
    text += page.extract_text()

Check for Encryption First

Always check if a PDF is password-protected before processing:

from pypdf import PdfReader

reader = PdfReader("document.pdf")

if reader.is_encrypted:
    print("PDF is encrypted")
    # Try to decrypt with password
    if reader.decrypt("password"):
        print("Successfully decrypted")
    else:
        print("Failed to decrypt - wrong password")
        # Option 1: Ask user for password
        # Option 2: Use qpdf to remove password (if you have it)
        # qpdf --password=yourpassword --decrypt encrypted.pdf decrypted.pdf
else:
    print("PDF is not encrypted - proceed with processing")

For batch processing, skip or report encrypted files:

import os
from pypdf import PdfReader

for filename in os.listdir("pdfs/"):
    if filename.endswith(".pdf"):
        try:
            reader = PdfReader(f"pdfs/{filename}")
            if reader.is_encrypted:
                print(f"SKIP: {filename} - encrypted")
                continue
            # Process the PDF...
        except Exception as e:
            print(f"ERROR: {filename} - {e}")

Python Libraries

pypdf - Basic Operations

Merge PDFs

from pypdf import PdfWriter, PdfReader

writer = PdfWriter()
for pdf_file in ["doc1.pdf", "doc2.pdf", "doc3.pdf"]:
    reader = PdfReader(pdf_file)
    for page in reader.pages:
        writer.add_page(page)

with open("merged.pdf", "wb") as output:
    writer.write(output)

Split PDF

reader = PdfReader("input.pdf")
for i, page in enumerate(reader.pages):
    writer = PdfWriter()
    writer.add_page(page)
    with open(f"page_{i+1}.pdf", "wb") as output:
        writer.write(output)

Extract Metadata

reader = PdfReader("document.pdf")
meta = reader.metadata
print(f"Title: {meta.title}")
print(f"Author: {meta.author}")
print(f"Subject: {meta.subject}")
print(f"Creator: {meta.creator}")

Rotate Pages

reader = PdfReader("input.pdf")
writer = PdfWriter()

page = reader.pages[0]
page.rotate(90)  # Rotate 90 degrees clockwise
writer.add_page(page)

with open("rotated.pdf", "wb") as output:
    writer.write(output)

pdfplumber - Text and Table Extraction

Extract Text with Layout

import pdfplumber

with pdfplumber.open("document.pdf") as pdf:
    for page in pdf.pages:
        text = page.extract_text()
        print(text)

Extract Tables

with pdfplumber.open("document.pdf") as pdf:
    for i, page in enumerate(pdf.pages):
        tables = page.extract_tables()
        for j, table in enumerate(tables):
            print(f"Table {j+1} on page {i+1}:")
            for row in table:
                print(row)

Advanced Table Extraction

import pandas as pd

with pdfplumber.open("document.pdf") as pdf:
    all_tables = []
    for page in pdf.pages:
        tables = page.extract_tables()
        for table in tables:
            if table:  # Check if table is not empty
                df = pd.DataFrame(table[1:], columns=table[0])
                all_tables.append(df)

# Combine all tables
if all_tables:
    combined_df = pd.concat(all_tables, ignore_index=True)
    combined_df.to_excel("extracted_tables.xlsx", index=False)

reportlab - Create PDFs

Basic PDF Creation

from reportlab.lib.pagesizes import letter
from reportlab.pdfgen import canvas

c = canvas.Canvas("hello.pdf", pagesize=letter)
width, height = letter

# Add text
c.drawString(100, height - 100, "Hello World!")
c.drawString(100, height - 120, "This is a PDF created with reportlab")

# Add a line
c.line(100, height - 140, 400, height - 140)

# Save
c.save()

Create PDF with Multiple Pages

from reportlab.lib.pagesizes import letter
from reportlab.platypus import SimpleDocTemplate, Paragraph, Spacer, PageBreak
from reportlab.lib.styles import getSampleStyleSheet

doc = SimpleDocTemplate("report.pdf", pagesize=letter)
styles = getSampleStyleSheet()
story = []

# Add content
title = Paragraph("Report Title", styles['Title'])
story.append(title)
story.append(Spacer(1, 12))

body = Paragraph("This is the body of the report. " * 20, styles['Normal'])
story.append(body)
story.append(PageBreak())

# Page 2
story.append(Paragraph("Page 2", styles['Heading1']))
story.append(Paragraph("Content for page 2", styles['Normal']))

# Build PDF
doc.build(story)

Command-Line Tools

pdftotext (poppler-utils)

# Extract text
pdftotext input.pdf output.txt

# Extract text preserving layout
pdftotext -layout input.pdf output.txt

# Extract specific pages
pdftotext -f 1 -l 5 input.pdf output.txt  # Pages 1-5

qpdf

# Merge PDFs
qpdf --empty --pages file1.pdf file2.pdf -- merged.pdf

# Split pages
qpdf input.pdf --pages . 1-5 -- pages1-5.pdf
qpdf input.pdf --pages . 6-10 -- pages6-10.pdf

# Rotate pages
qpdf input.pdf output.pdf --rotate=+90:1  # Rotate page 1 by 90 degrees

# Remove password
qpdf --password=mypassword --decrypt encrypted.pdf decrypted.pdf

pdftk (if available)

# Merge
pdftk file1.pdf file2.pdf cat output merged.pdf

# Split
pdftk input.pdf burst

# Rotate
pdftk input.pdf rotate 1east output rotated.pdf

Common Tasks

Extract Text from Scanned PDFs

# Requires: pip install pytesseract pdf2image
import pytesseract
from pdf2image import convert_from_path

# Convert PDF to images
images = convert_from_path('scanned.pdf')

# OCR each page
text = ""
for i, image in enumerate(images):
    text += f"Page {i+1}:\n"
    text += pytesseract.image_to_string(image)
    text += "\n\n"

print(text)

Add Watermark

from pypdf import PdfReader, PdfWriter

# Create watermark (or load existing)
watermark = PdfReader("watermark.pdf").pages[0]

# Apply to all pages
reader = PdfReader("document.pdf")
writer = PdfWriter()

for page in reader.pages:
    page.merge_page(watermark)
    writer.add_page(page)

with open("watermarked.pdf", "wb") as output:
    writer.write(output)

Extract Images

# Using pdfimages (poppler-utils)
pdfimages -j input.pdf output_prefix

# This extracts all images as output_prefix-000.jpg, output_prefix-001.jpg, etc.

Password Protection

from pypdf import PdfReader, PdfWriter

reader = PdfReader("input.pdf")
writer = PdfWriter()

for page in reader.pages:
    writer.add_page(page)

# Add password
writer.encrypt("userpassword", "ownerpassword")

with open("encrypted.pdf", "wb") as output:
    writer.write(output)

Quick Reference

Troubleshooting

Corrupted or Invalid PDFs

Symptom: Errors like "EOF marker not found", "Invalid PDF structure", or "Cannot read PDF"

Solution: Use qpdf to repair the PDF first:

qpdf --check input.pdf  # Check for errors
qpdf input.pdf repaired.pdf  # Attempt repair

If qpdf can't repair it, try:

gs -o repaired.pdf -sDEVICE=pdfwrite -dPDFSETTINGS=/prepress input.pdf

Missing Dependencies

Symptom: ModuleNotFoundError or command not found

Python libraries:

pip install pypdf pdfplumber reportlab pytesseract pdf2image pandas

Command-line tools (macOS):

brew install poppler qpdf tesseract

Command-line tools (Linux):

sudo apt-get install poppler-utils qpdf tesseract-ocr

Memory Issues with Large PDFs

Symptom: Process killed, memory errors, or system slowdown when processing PDFs >100 pages

Solution: Process in chunks:

from pypdf import PdfReader, PdfWriter

reader = PdfReader("large.pdf")
chunk_size = 50  # Process 50 pages at a time

for i in range(0, len(reader.pages), chunk_size):
    writer = PdfWriter()
    for page_num in range(i, min(i + chunk_size, len(reader.pages))):
        writer.add_page(reader.pages[page_num])

    with open(f"chunk_{i//chunk_size + 1}.pdf", "wb") as output:
        writer.write(output)

Or extract text page-by-page:

with pdfplumber.open("large.pdf") as pdf:
    for i, page in enumerate(pdf.pages):
        text = page.extract_text()
        # Process or save immediately
        with open(f"page_{i+1}.txt", "w") as f:
            f.write(text)

Permission Errors

Symptom: "PermissionError" or "Access denied"

Causes:

1. PDF is password-protected (see "Check for Encryption First" section)
2. File is open in another application (close it)
3. Insufficient file system permissions

Solution:

# Check file permissions
ls -la file.pdf

# Make readable
chmod 644 file.pdf

Text Extraction Returns Empty or Garbled Text

Symptom: extract_text() returns empty string or gibberish

Causes:

1. PDF is scanned (image-based) - use OCR
2. PDF uses non-standard fonts or encoding
3. Text is embedded as images

Solution for scanned PDFs:

# Use pytesseract for OCR (see "Extract Text from Scanned PDFs" section)
from pdf2image import convert_from_path
import pytesseract

images = convert_from_path('scanned.pdf')
text = pytesseract.image_to_string(images[0])

Solution for non-standard encoding: Try pdfplumber instead of pypdf:

import pdfplumber

with pdfplumber.open("document.pdf") as pdf:
    text = pdf.pages[0].extract_text()

Or use command-line tool:

pdftotext -layout document.pdf output.txt

Next Steps

• For advanced pypdfium2 usage, see reference.md
• For JavaScript libraries (pdf-lib), see reference.md
• If you need to fill out a PDF form, follow the instructions in forms.md
• For troubleshooting guides, see reference.md