Format-Specific Extraction Workflows
Office XML (DOCX/PPTX/ODT)
ZIP archive → SecurityBudget → XML parsing → Text + tables + metadata
let budget = SecurityBudget::from_config(config); (extractors/security.rs), plus
config.security_limits…max_files_in_archive as the member cap. The Office path does
not use ZipBombValidator — that is the archive/iWork/HWPX path.- Open with
zip::ZipArchive::new(cursor) and read the parts
(word/document.xml, ppt/slides/*.xml, content.xml).
- Parse with
quick-xml::Reader (streaming), threading &mut budget through the recursive
walkers so a hostile document exhausts a budget instead of memory.
- Metadata via
crate::extraction::office_metadata — see the helper table below. There is no
extract_metadata().
- See
extractors/docx.rs, extractors/pptx.rs, extractors/odt.rs.
PDF
Bytes → xberg_native_pdf → Per-page text + OCR fallback → Tables → Metadata
xberg_native_pdf::PdfDocument::from_bytes(content.to_vec())? — the engine takes an owned
Vec<u8>, not a slice.- OCR is forced by
config.force_ocr (whole document) or config.force_ocr_pages
(Option<Vec<u32>>); otherwise pages with no extractable text route to OCR.
config.pages: Option<PageConfig> controls per-page output — it does not gate tables.- Feature-gated
#[cfg(feature = "pdf")]; the backend is PdfConfig.backend
(native default, pdfium behind pdf-pdfium).
- See
extractors/pdf/mod.rs.
Archives (ZIP/TAR/7z/GZIP)
ZipBombValidator → per-format metadata → per-format text content
ZipBombValidator::new(limits).validate(&mut archive)? before any extraction.- Metadata and content come from per-format helpers in
extraction/archive/:
extract_{zip,tar,7z,gzip}_metadata, extract_{zip,tar,7z,gzip}_text_content,
extract_{zip,tar,7z}_file_bytes. There is no build_archive_result().
- See
extractors/archive.rs, extraction/archive/{zip,tar,sevenz,gzip}.rs.
Structured Text (JSON/YAML/TOML/XML)
Single StructuredExtractor covers several MIME types: parse with the format library,
pretty-print to text. See extractors/structured.rs.
Email (EML/MSG/PST)
Parse headers → extract body (text/html) → process attachments. Message-in-message nesting is
bounded by the SecurityBudget's SecurityLimits-derived DepthValidator, the same counter
every other format uses. See extraction/email.rs, extractors/email.rs, extractors/pst.rs.
Common Helpers
| Helper |
Location |
extract_core_properties() |
extraction/office_metadata/core_properties.rs |
extract_custom_properties() |
extraction/office_metadata/custom_properties.rs |
extract_{docx,xlsx,pptx}_app_properties() |
extraction/office_metadata/app_properties.rs |
extract_odt_properties() |
extraction/office_metadata/odt_properties.rs |
cells_to_markdown() |
extraction/markdown.rs (pub(crate)) |
SecurityBudget, SecurityLimits |
extractors/security.rs |
ZipBombValidator, DepthValidator |
extractors/security.rs |
StringGrowthValidator |
extractors/security.rs |
The security types are pub(crate): in-crate extractors can use them, out-of-crate plugin
authors cannot.
Adding a New Format
- Add one
FormatEntry to the FORMATS registry in core/mime.rs. EXT_TO_MIME and
SUPPORTED_MIME_TYPES are derived from it — do not hand-edit either. See
mime-detection-routing for the full procedure, including the count assertion to bump.
- Create an extractor implementing
InternalDocumentExtractor (not DocumentExtractor).
- Set
supported_mime_types() and priority() (default 50).
- Register in
extractors/mod.rs → register_default_extractors().
- Feature-gate if optional:
#[cfg(feature = "my-format")].
- Apply
SecurityBudget / SecurityLimits to any user-supplied content.
- Add
#[cfg_attr(alef, alef(skip))] to the extractor struct or the binding regen aborts.
- Add tests with fixture files (see the
test-corpus skill for where fixtures come from).
1---2name: format-specific-extraction3description: Format-specific document extraction workflows4---5
6# Format-Specific Extraction Workflows
7
8## Office XML (DOCX/PPTX/ODT)
9
10```text
11ZIP archive → SecurityBudget → XML parsing → Text + tables + metadata
12```
13
141. `let budget = SecurityBudget::from_config(config);` (`extractors/security.rs`), plus
15 `config.security_limits…max_files_in_archive` as the member cap. The Office path does
16 **not** use `ZipBombValidator` — that is the archive/iWork/HWPX path.
172. Open with `zip::ZipArchive::new(cursor)` and read the parts
18 (`word/document.xml`, `ppt/slides/*.xml`, `content.xml`).
193. Parse with `quick-xml::Reader` (streaming), threading `&mut budget` through the recursive
20 walkers so a hostile document exhausts a budget instead of memory.
214. Metadata via `crate::extraction::office_metadata` — see the helper table below. There is no
22 `extract_metadata()`.
235. See `extractors/docx.rs`, `extractors/pptx.rs`, `extractors/odt.rs`.
24
25## PDF
26
27```text
28Bytes → xberg_native_pdf → Per-page text + OCR fallback → Tables → Metadata
29```
30
311. `xberg_native_pdf::PdfDocument::from_bytes(content.to_vec())?` — the engine takes an owned
32 `Vec<u8>`, not a slice.
332. OCR is forced by `config.force_ocr` (whole document) or `config.force_ocr_pages`
34 (`Option<Vec<u32>>`); otherwise pages with no extractable text route to OCR.
353. `config.pages: Option<PageConfig>` controls per-page output — it does not gate tables.
364. Feature-gated `#[cfg(feature = "pdf")]`; the backend is `PdfConfig.backend`
37 (`native` default, `pdfium` behind `pdf-pdfium`).
385. See `extractors/pdf/mod.rs`.
39
40## Archives (ZIP/TAR/7z/GZIP)
41
42```text
43ZipBombValidator → per-format metadata → per-format text content
44```
45
461. `ZipBombValidator::new(limits).validate(&mut archive)?` before any extraction.
472. Metadata and content come from per-format helpers in `extraction/archive/`:
48 `extract_{zip,tar,7z,gzip}_metadata`, `extract_{zip,tar,7z,gzip}_text_content`,
49 `extract_{zip,tar,7z}_file_bytes`. There is no `build_archive_result()`.
503. See `extractors/archive.rs`, `extraction/archive/{zip,tar,sevenz,gzip}.rs`.
51
52## Structured Text (JSON/YAML/TOML/XML)
53
54Single `StructuredExtractor` covers several MIME types: parse with the format library,
55pretty-print to text. See `extractors/structured.rs`.
56
57## Email (EML/MSG/PST)
58
59Parse headers → extract body (text/html) → process attachments. Message-in-message nesting is
60bounded by the `SecurityBudget`'s `SecurityLimits`-derived `DepthValidator`, the same counter
61every other format uses. See `extraction/email.rs`, `extractors/email.rs`, `extractors/pst.rs`.
62
63## Common Helpers
64
65| Helper | Location |
66| --- | --- |
67| `extract_core_properties()` | `extraction/office_metadata/core_properties.rs` |
68| `extract_custom_properties()` | `extraction/office_metadata/custom_properties.rs` |
69| `extract_{docx,xlsx,pptx}_app_properties()` | `extraction/office_metadata/app_properties.rs` |
70| `extract_odt_properties()` | `extraction/office_metadata/odt_properties.rs` |
71| `cells_to_markdown()` | `extraction/markdown.rs` (`pub(crate)`) |
72| `SecurityBudget`, `SecurityLimits` | `extractors/security.rs` |
73| `ZipBombValidator`, `DepthValidator` | `extractors/security.rs` |
74| `StringGrowthValidator` | `extractors/security.rs` |
75
76The security types are `pub(crate)`: in-crate extractors can use them, out-of-crate plugin
77authors cannot.
78
79## Adding a New Format
80
811. Add one `FormatEntry` to the `FORMATS` registry in `core/mime.rs`. `EXT_TO_MIME` and
82 `SUPPORTED_MIME_TYPES` are derived from it — do not hand-edit either. See
83 `mime-detection-routing` for the full procedure, including the count assertion to bump.
842. Create an extractor implementing `InternalDocumentExtractor` (not `DocumentExtractor`).
853. Set `supported_mime_types()` and `priority()` (default 50).
864. Register in `extractors/mod.rs → register_default_extractors()`.
875. Feature-gate if optional: `#[cfg(feature = "my-format")]`.
886. Apply `SecurityBudget` / `SecurityLimits` to any user-supplied content.
897. Add `#[cfg_attr(alef, alef(skip))]` to the extractor struct or the binding regen aborts.
908. Add tests with fixture files (see the `test-corpus` skill for where fixtures come from).
gabrielmoreira