Extraction Pipeline Patterns
Kreuzberg's format detection -> extraction -> fallback orchestration for 75+ file formats
Core Pipeline Architecture
The extraction pipeline (crates/kreuzberg/src/core/pipeline.rs, crates/kreuzberg/src/extraction/) orchestrates:
- Format Detection - MIME type inference + extension validation -> select appropriate extractor
- Intelligent Extraction - Route to format-specific extractors (PDF, DOCX, Excel, HTML, images, archives, etc.)
- Fallback Strategies - Password-protected PDFs, OCR for images, nested archive handling, corrupted file recovery
- Post-Processing Pipeline - Validators, quality processing, chunking, custom hooks (see
core/pipeline.rs)
Format Detection Strategy
Location: crates/kreuzberg/src/core/mime.rs, crates/kreuzberg/src/core/formats.rs
Pattern: detect via magic bytes, validate extension alignment (prevent spoofing), route to extractor. Multiple extractors for same format -> choose highest confidence/specificity.
// Pseudocode: core/mime.rs
match (magic_bytes(content), extension) {
(Some(fmt), Some(ext)) if aligned -> Ok(fmt),
(Some(fmt), Some(ext)) if misaligned -> Err(FormatMismatch),
(Some(fmt), None) -> Ok(fmt), // magic bytes only
(None, Some(ext)) -> Ok(from_extension(ext)),
_ -> Err(UnknownFormat),
}
Extraction Modules (75 Formats)
| Category |
Extractors |
Key Modules |
| Office |
DOCX, XLSX, XLSM, XLSB, XLS, PPTX, ODP, ODS |
extraction/{docx,excel,pptx}.rs |
| PDF |
Standard + encrypted, password attempts |
pdf/ subdirectory (13 files) |
| Images |
PNG, JPG, TIFF, WebP, JP2, SVG (OCR-enabled) |
extraction/image.rs + ocr/ |
| Web |
HTML, XHTML, XML, SVG (DOM parsing) |
extraction/html.rs (67KB - complex table handling) |
| Email |
EML, MSG (headers, body, attachments, threading) |
extraction/email.rs |
| Archives |
ZIP, TAR, GZ, 7Z (recursive extraction) |
extraction/archive.rs (31KB) |
| Markdown |
MD, TXT, RST, Org Mode, RTF |
extraction/markdown.rs |
| Academic |
LaTeX, BibTeX, JATS, Jupyter, DocBook |
extraction/{structured,xml}.rs |
Extraction Dispatcher
// Pseudocode: extraction/mod.rs
let format = detect_format(source.bytes, source.extension);
let result = match format {
Pdf -> extract_pdf(source, config),
Docx -> extract_docx(source, config),
Image -> extract_image_with_ocr_fallback(source, config),
Archive -> extract_archive_recursive(source, config),
_ -> extract_with_plugin(format, source, config),
};
run_pipeline(result, config) // post-processing always runs
Fallback Strategies
- Password-Protected PDFs: Try primary password -> secondary password list -> return
is_encrypted=true in metadata on failure
- OCR Fallback: If image text extraction confidence < threshold, trigger OCR backend; return both results with scores
- Nested Archives: Recursive extraction with configurable depth limit; flatten or preserve hierarchy
- Corrupted File Recovery: Stream-based parsing, emit content up to error point, include error location in metadata
Configuration Integration
Location: crates/kreuzberg/src/core/config.rs, crates/kreuzberg/src/core/config_validation.rs
ExtractionConfig holds format-specific configs (pdf, image, html, office), fallback orchestration (fallback), and post-processing (postprocessor, chunking, keywords). See struct definition in config.rs.
Plugin System Integration
Location: crates/kreuzberg/src/plugins/
- CustomExtractor: Override built-in format extractors
- PostProcessor: Modify results after extraction (Early/Middle/Late stages)
- Validator: Fail-fast validation (e.g., minimum text length)
- OCRBackend: Swap OCR engine
Plugin registry loaded at startup, cached for zero-cost lookup.
Feature Flag Strategy
Location: Cargo.toml (workspace), crates/kreuzberg/Cargo.toml, FEATURE_MATRIX.md
20+ features across 9 language bindings. Key feature groups:
| Group |
Features |
Notes |
| OCR |
tesseract (default), tesseract-static, ocr-minimal |
Mutually exclusive recommendation |
| Formats |
pdf, pdf-minimal, office, office-minimal |
|
| AI/ML |
embeddings (requires ONNX), keywords-yake, keywords-rake, language-detection |
|
| Server |
api (Axum), mcp, tokio-runtime, lite-runtime |
|
| Bindings |
python-bindings, ruby-bindings, php-bindings, node-bindings, wasm |
|
Conditional compilation: modules gated with #[cfg(feature = "...")]. Runtime validate_config() warns if requested feature not compiled in.
Feature Flag Critical Rules
- Never mix conflicting features - e.g.,
ocr-minimal + tesseract should error at compile time
- Always provide feature diagnostics - Config validation must warn if feature unavailable
- Default to maximum feature set - Unless embedded/minimal specifically requested
- Test all feature combinations - Matrix testing in CI catches regressions
- WASM incompatible with embeddings, keywords, OCR
Critical Rules
- Always use format detection before routing to extractors (prevent confusion attacks)
- Stream-based parsing for PDFs/archives to handle multi-GB files
- Post-pipeline is mandatory: All extraction results flow through
run_pipeline() for validators/hooks
- Plugin overrides are order-dependent: Plugins registered first take priority
- Fallback timeouts: Set reasonable OCR/archive extraction timeouts (config-driven)
- Metadata preservation: Include format detection confidence, extraction method used, any fallbacks applied
Related Skills
- ocr-backend-management - OCR engine selection and image preprocessing
- chunking-embeddings - Post-extraction text splitting with FastEmbed
- api-server-mcp - Axum endpoint for extraction pipeline exposure and MCP server
1---2description: "Document extraction pipeline architecture and patterns"3name: extraction-pipeline-patterns4priority: critical5metadata:6 source: https://skillmd.com/skills/zuoguyoupan2023/extraction-pipeline-patterns7---89# Extraction Pipeline Patterns1011**Kreuzberg's format detection -> extraction -> fallback orchestration for 75+ file formats**1213## Core Pipeline Architecture1415The extraction pipeline (`crates/kreuzberg/src/core/pipeline.rs`, `crates/kreuzberg/src/extraction/`) orchestrates:16171. **Format Detection** - MIME type inference + extension validation -> select appropriate extractor182. **Intelligent Extraction** - Route to format-specific extractors (PDF, DOCX, Excel, HTML, images, archives, etc.)193. **Fallback Strategies** - Password-protected PDFs, OCR for images, nested archive handling, corrupted file recovery204. **Post-Processing Pipeline** - Validators, quality processing, chunking, custom hooks (see `core/pipeline.rs`)2122## Format Detection Strategy2324**Location**: `crates/kreuzberg/src/core/mime.rs`, `crates/kreuzberg/src/core/formats.rs`2526Pattern: detect via magic bytes, validate extension alignment (prevent spoofing), route to extractor. Multiple extractors for same format -> choose highest confidence/specificity.2728```rust29// Pseudocode: core/mime.rs30match (magic_bytes(content), extension) {31 (Some(fmt), Some(ext)) if aligned -> Ok(fmt),32 (Some(fmt), Some(ext)) if misaligned -> Err(FormatMismatch),33 (Some(fmt), None) -> Ok(fmt), // magic bytes only34 (None, Some(ext)) -> Ok(from_extension(ext)),35 _ -> Err(UnknownFormat),36}37```3839## Extraction Modules (75 Formats)4041| Category | Extractors | Key Modules |42| ------------ | ------------------------------------------------ | ---------------------------------------------------- |43| **Office** | DOCX, XLSX, XLSM, XLSB, XLS, PPTX, ODP, ODS | `extraction/{docx,excel,pptx}.rs` |44| **PDF** | Standard + encrypted, password attempts | `pdf/` subdirectory (13 files) |45| **Images** | PNG, JPG, TIFF, WebP, JP2, SVG (OCR-enabled) | `extraction/image.rs` + `ocr/` |46| **Web** | HTML, XHTML, XML, SVG (DOM parsing) | `extraction/html.rs` (67KB - complex table handling) |47| **Email** | EML, MSG (headers, body, attachments, threading) | `extraction/email.rs` |48| **Archives** | ZIP, TAR, GZ, 7Z (recursive extraction) | `extraction/archive.rs` (31KB) |49| **Markdown** | MD, TXT, RST, Org Mode, RTF | `extraction/markdown.rs` |50| **Academic** | LaTeX, BibTeX, JATS, Jupyter, DocBook | `extraction/{structured,xml}.rs` |5152## Extraction Dispatcher5354```rust55// Pseudocode: extraction/mod.rs56let format = detect_format(source.bytes, source.extension);57let result = match format {58 Pdf -> extract_pdf(source, config),59 Docx -> extract_docx(source, config),60 Image -> extract_image_with_ocr_fallback(source, config),61 Archive -> extract_archive_recursive(source, config),62 _ -> extract_with_plugin(format, source, config),63};64run_pipeline(result, config) // post-processing always runs65```6667## Fallback Strategies6869- **Password-Protected PDFs**: Try primary password -> secondary password list -> return `is_encrypted=true` in metadata on failure70- **OCR Fallback**: If image text extraction confidence < threshold, trigger OCR backend; return both results with scores71- **Nested Archives**: Recursive extraction with configurable depth limit; flatten or preserve hierarchy72- **Corrupted File Recovery**: Stream-based parsing, emit content up to error point, include error location in metadata7374## Configuration Integration7576**Location**: `crates/kreuzberg/src/core/config.rs`, `crates/kreuzberg/src/core/config_validation.rs`7778`ExtractionConfig` holds format-specific configs (`pdf`, `image`, `html`, `office`), fallback orchestration (`fallback`), and post-processing (`postprocessor`, `chunking`, `keywords`). See struct definition in `config.rs`.7980## Plugin System Integration8182**Location**: `crates/kreuzberg/src/plugins/`8384- **CustomExtractor**: Override built-in format extractors85- **PostProcessor**: Modify results after extraction (Early/Middle/Late stages)86- **Validator**: Fail-fast validation (e.g., minimum text length)87- **OCRBackend**: Swap OCR engine8889Plugin registry loaded at startup, cached for zero-cost lookup.9091## Feature Flag Strategy9293**Location**: `Cargo.toml` (workspace), `crates/kreuzberg/Cargo.toml`, `FEATURE_MATRIX.md`949520+ features across 9 language bindings. Key feature groups:9697| Group | Features | Notes |98| -------- | ------------------------------------------------------------------------------------ | --------------------------------- |99| OCR | `tesseract` (default), `tesseract-static`, `ocr-minimal` | Mutually exclusive recommendation |100| Formats | `pdf`, `pdf-minimal`, `office`, `office-minimal` | |101| AI/ML | `embeddings` (requires ONNX), `keywords-yake`, `keywords-rake`, `language-detection` | |102| Server | `api` (Axum), `mcp`, `tokio-runtime`, `lite-runtime` | |103| Bindings | `python-bindings`, `ruby-bindings`, `php-bindings`, `node-bindings`, `wasm` | |104105Conditional compilation: modules gated with `#[cfg(feature = "...")]`. Runtime `validate_config()` warns if requested feature not compiled in.106107### Feature Flag Critical Rules1081091. **Never mix conflicting features** - e.g., `ocr-minimal` + `tesseract` should error at compile time1102. **Always provide feature diagnostics** - Config validation must warn if feature unavailable1113. **Default to maximum feature set** - Unless embedded/minimal specifically requested1124. **Test all feature combinations** - Matrix testing in CI catches regressions1135. **WASM incompatible** with embeddings, keywords, OCR114115## Critical Rules1161171. **Always use format detection** before routing to extractors (prevent confusion attacks)1182. **Stream-based parsing** for PDFs/archives to handle multi-GB files1193. **Post-pipeline is mandatory**: All extraction results flow through `run_pipeline()` for validators/hooks1204. **Plugin overrides are order-dependent**: Plugins registered first take priority1215. **Fallback timeouts**: Set reasonable OCR/archive extraction timeouts (config-driven)1226. **Metadata preservation**: Include format detection confidence, extraction method used, any fallbacks applied123124## Related Skills125126- **ocr-backend-management** - OCR engine selection and image preprocessing127- **chunking-embeddings** - Post-extraction text splitting with FastEmbed128- **api-server-mcp** - Axum endpoint for extraction pipeline exposure and MCP server