PDF Parse

PDF Parsing

Dual-path extraction pipeline. Selects tool based on PDF type.

Tool overview

Step 1 — Ensure tools are installed

xberg

# Check
xberg --version

# Install (macOS)
brew tap xberg-io/tap
brew install xberg


# Install (Linux / no brew)
# Download binary from https://github.com/xberg-io/xberg/releases
# or via Docker: docker pull ghcr.io/xberg-io/xberg:latest

marker-pdf

marker-pdf is invoked via uvx — no persistent install needed.

# Verify uvx is available
uvx --version

# Test marker-pdf is resolvable
uvx marker-pdf --version

If uvx is missing, install uv first: curl -LsSf https://astral.sh/uv/install.sh | sh

pdftotext (fallback)

# macOS
brew install poppler

# Debian/Ubuntu
apt-get install poppler-utils

Step 2 — Detect PDF type

# xberg detect — confirms file is a valid PDF (returns MIME type only)
xberg detect "<path/to/file.pdf>"

# To distinguish machine-readable vs scanned, use pdfinfo
pdfinfo "<path/to/file.pdf>"

Known limitation: xberg detect returns application/pdf for all PDFs — it does NOT distinguish machine-readable from scanned. Use it only to confirm the file is a valid PDF.

Decision rule:

• pdfinfo shows Pages: N and text content present (non-zero Characters:) → Step 3A (xberg)
• pdfinfo shows Characters: 0 or < 100 → Step 3B (marker-pdf, likely scanned)
• xberg unavailable → Step 3B (marker-pdf) or Step 3C (pdftotext fallback)
• Neither tool available → Step 3C (pdftotext fallback)

Step 3 — Extract

3A — Machine-readable PDF → xberg

Warning: xberg writes WARN-level log lines to stdout, not stderr. Using > redirection captures them inside the output file. Always use --output to avoid polluted Markdown.

# Markdown output (default, RAG-friendly)
xberg extract "<path/to/file.pdf>" --content-format markdown --output "<output/file.md>"

# JSON output (structured tree)
xberg extract "<path/to/file.pdf>" --content-format json --output "<output/file.json>"

# With page markers (useful for large PDFs — adds <!-- page N --> separators)
xberg extract "<path/to/file.pdf>" --content-format markdown --page-markers true --output "<output/file.md>"

# Batch extraction
xberg batch "<input_dir/" --content-format markdown --output-dir "<output_dir/"

# If --output is unavailable and stdout redirect is the only option, strip WARN lines after:
xberg extract "<path/to/file.pdf>" --content-format markdown > /tmp/raw.md
grep -v '^WARN' /tmp/raw.md > "<output/file.md>"

3B — Scanned / image PDF → marker-pdf

# Standard scanned PDF
uvx marker-pdf marker_single "<path/to/file.pdf>" --output_dir "<output_dir/"

# Academic paper — LLM-enhanced (best quality, slower, requires OPENAI_API_KEY or compatible)
uvx marker-pdf marker_single "<path/to/file.pdf>" --use_llm --force_ocr --output_dir "<output_dir/"

# Force OCR without LLM
uvx marker-pdf marker_single "<path/to/file.pdf>" --force_ocr --output_dir "<output_dir/"

# Large scanned PDF — split by page range to avoid OOM
uvx marker-pdf marker_single "<path/to/file.pdf>" --page_range "0-49" --output_dir "<output_dir/part1/"
uvx marker-pdf marker_single "<path/to/file.pdf>" --page_range "50-99" --output_dir "<output_dir/part2/"

marker-pdf produces a subdirectory per document containing <slug>.md and images.

3C — Fallback → pdftotext

pdftotext "<path/to/file.pdf>" "<output/file.txt>"

# Preserve layout (tables, columns)
pdftotext -layout "<path/to/file.pdf>" "<output/file.txt>"

Step 4 — Verify output

# Quick sanity check — word count and line count
wc -w -l "<output/file.md>"

# Check for garbled content (non-UTF8 sequences)
file "<output/file.md>"

If word count is suspiciously low (< 200 words for a multi-page PDF), retry with the other tool path.

Tool maintenance

Update xberg

brew upgrade xberg-io/tap/xberg
xberg --version

Update marker-pdf

uvx always pulls the latest version on first invocation per session. To force refresh of cached venv:

uvx --reinstall marker-pdf marker_single --version

Update pdftotext

# macOS
brew upgrade poppler

# Debian/Ubuntu
apt-get upgrade poppler-utils

Diagnose xberg

xberg doctor

Reports missing system deps (tesseract, pdfium), OCR language packs, and version info.

Clean xberg cache

# Show cache stats
xberg cache stats

# Clear all cached extractions
xberg cache clear

# Clear cache for one file
xberg cache clear "<path/to/file.pdf>"

Uninstall

# xberg
brew uninstall xberg

# marker-pdf (no persistent install — clear uvx cache)
uvx cache clean marker-pdf
# or wipe all uvx caches
rm -rf "$(uvx env --path 2>/dev/null || echo ~/.cache/uv/tool-environments)"

Quick decision table