Format-Specific Extraction Workflows
Office XML (DOCX/PPTX/ODT)
ZIP archive → SecurityBudget → XML parsing → Text + tables + metadata
let budget = SecurityBudget::from_config(config); (extractors/security.rs), plus
config.security_limits…max_files_in_archive as the member cap. The Office path does
not use ZipBombValidator — that is the archive/iWork/HWPX path.
- Open with
zip::ZipArchive::new(cursor) and read the parts
(word/document.xml, ppt/slides/*.xml, content.xml).
- Parse with
quick-xml::Reader (streaming), threading &mut budget through the recursive
walkers so a hostile document exhausts a budget instead of memory.
- Metadata via
crate::extraction::office_metadata — see the helper table below. There is no
extract_metadata().
- See
extractors/docx.rs, extractors/pptx.rs, extractors/odt.rs.
PDF
Bytes → xberg_native_pdf → Per-page text + OCR fallback → Tables → Metadata
xberg_native_pdf::PdfDocument::from_bytes(content.to_vec())? — the engine takes an owned
Vec<u8>, not a slice.
- OCR is forced by
config.force_ocr (whole document) or config.force_ocr_pages
(Option<Vec<u32>>); otherwise pages with no extractable text route to OCR.
config.pages: Option<PageConfig> controls per-page output — it does not gate tables.
- Feature-gated
#[cfg(feature = "pdf")]; the backend is PdfConfig.backend
(native default, pdfium behind pdf-pdfium).
- See
extractors/pdf/mod.rs.
Archives (ZIP/TAR/7z/GZIP)
ZipBombValidator → per-format metadata → per-format text content
ZipBombValidator::new(limits).validate(&mut archive)? before any extraction.
- Metadata and content come from per-format helpers in
extraction/archive/:
extract_{zip,tar,7z,gzip}_metadata, extract_{zip,tar,7z,gzip}_text_content,
extract_{zip,tar,7z}_file_bytes. There is no build_archive_result().
- See
extractors/archive.rs, extraction/archive/{zip,tar,sevenz,gzip}.rs.
Structured Text (JSON/YAML/TOML/XML)
Single StructuredExtractor covers several MIME types: parse with the format library,
pretty-print to text. See extractors/structured.rs.
Email (EML/MSG/PST)
Parse headers → extract body (text/html) → process attachments. Message-in-message nesting is
bounded by the SecurityBudget's SecurityLimits-derived DepthValidator, the same counter
every other format uses. See extraction/email.rs, extractors/email.rs, extractors/pst.rs.
Common Helpers
| Helper |
Location |
extract_core_properties() |
extraction/office_metadata/core_properties.rs |
extract_custom_properties() |
extraction/office_metadata/custom_properties.rs |
extract_{docx,xlsx,pptx}_app_properties() |
extraction/office_metadata/app_properties.rs |
extract_odt_properties() |
extraction/office_metadata/odt_properties.rs |
cells_to_markdown() |
extraction/markdown.rs (pub(crate)) |
SecurityBudget, SecurityLimits |
extractors/security.rs |
ZipBombValidator, DepthValidator |
extractors/security.rs |
StringGrowthValidator |
extractors/security.rs |
The security types are pub(crate): in-crate extractors can use them, out-of-crate plugin
authors cannot.
Adding a New Format
- Add one
FormatEntry to the FORMATS registry in core/mime.rs. EXT_TO_MIME and
SUPPORTED_MIME_TYPES are derived from it — do not hand-edit either. See
mime-detection-routing for the full procedure, including the count assertion to bump.
- Create an extractor implementing
InternalDocumentExtractor (not DocumentExtractor).
- Set
supported_mime_types() and priority() (default 50).
- Register in
extractors/mod.rs → register_default_extractors().
- Feature-gate if optional:
#[cfg(feature = "my-format")].
- Apply
SecurityBudget / SecurityLimits to any user-supplied content.
- Add
#[cfg_attr(alef, alef(skip))] to the extractor struct or the binding regen aborts.
- Add tests with fixture files (see the
test-corpus skill for where fixtures come from).
1---
2name: format-specific-extraction
3description: "Format-specific document extraction workflows"
4priority: high
5metadata:
6 source: https://skillmd.com/skills/gabrielmoreira/format-specific-extraction
7---
8
9# Format-Specific Extraction Workflows
10
11## Office XML (DOCX/PPTX/ODT)
12
13```text
14ZIP archive → SecurityBudget → XML parsing → Text + tables + metadata
15```
16
171. `let budget = SecurityBudget::from_config(config);` (`extractors/security.rs`), plus
18 `config.security_limits…max_files_in_archive` as the member cap. The Office path does
19 **not** use `ZipBombValidator` — that is the archive/iWork/HWPX path.
202. Open with `zip::ZipArchive::new(cursor)` and read the parts
21 (`word/document.xml`, `ppt/slides/*.xml`, `content.xml`).
223. Parse with `quick-xml::Reader` (streaming), threading `&mut budget` through the recursive
23 walkers so a hostile document exhausts a budget instead of memory.
244. Metadata via `crate::extraction::office_metadata` — see the helper table below. There is no
25 `extract_metadata()`.
265. See `extractors/docx.rs`, `extractors/pptx.rs`, `extractors/odt.rs`.
27
28## PDF
29
30```text
31Bytes → xberg_native_pdf → Per-page text + OCR fallback → Tables → Metadata
32```
33
341. `xberg_native_pdf::PdfDocument::from_bytes(content.to_vec())?` — the engine takes an owned
35 `Vec<u8>`, not a slice.
362. OCR is forced by `config.force_ocr` (whole document) or `config.force_ocr_pages`
37 (`Option<Vec<u32>>`); otherwise pages with no extractable text route to OCR.
383. `config.pages: Option<PageConfig>` controls per-page output — it does not gate tables.
394. Feature-gated `#[cfg(feature = "pdf")]`; the backend is `PdfConfig.backend`
40 (`native` default, `pdfium` behind `pdf-pdfium`).
415. See `extractors/pdf/mod.rs`.
42
43## Archives (ZIP/TAR/7z/GZIP)
44
45```text
46ZipBombValidator → per-format metadata → per-format text content
47```
48
491. `ZipBombValidator::new(limits).validate(&mut archive)?` before any extraction.
502. Metadata and content come from per-format helpers in `extraction/archive/`:
51 `extract_{zip,tar,7z,gzip}_metadata`, `extract_{zip,tar,7z,gzip}_text_content`,
52 `extract_{zip,tar,7z}_file_bytes`. There is no `build_archive_result()`.
533. See `extractors/archive.rs`, `extraction/archive/{zip,tar,sevenz,gzip}.rs`.
54
55## Structured Text (JSON/YAML/TOML/XML)
56
57Single `StructuredExtractor` covers several MIME types: parse with the format library,
58pretty-print to text. See `extractors/structured.rs`.
59
60## Email (EML/MSG/PST)
61
62Parse headers → extract body (text/html) → process attachments. Message-in-message nesting is
63bounded by the `SecurityBudget`'s `SecurityLimits`-derived `DepthValidator`, the same counter
64every other format uses. See `extraction/email.rs`, `extractors/email.rs`, `extractors/pst.rs`.
65
66## Common Helpers
67
68| Helper | Location |
69| --- | --- |
70| `extract_core_properties()` | `extraction/office_metadata/core_properties.rs` |
71| `extract_custom_properties()` | `extraction/office_metadata/custom_properties.rs` |
72| `extract_{docx,xlsx,pptx}_app_properties()` | `extraction/office_metadata/app_properties.rs` |
73| `extract_odt_properties()` | `extraction/office_metadata/odt_properties.rs` |
74| `cells_to_markdown()` | `extraction/markdown.rs` (`pub(crate)`) |
75| `SecurityBudget`, `SecurityLimits` | `extractors/security.rs` |
76| `ZipBombValidator`, `DepthValidator` | `extractors/security.rs` |
77| `StringGrowthValidator` | `extractors/security.rs` |
78
79The security types are `pub(crate)`: in-crate extractors can use them, out-of-crate plugin
80authors cannot.
81
82## Adding a New Format
83
841. Add one `FormatEntry` to the `FORMATS` registry in `core/mime.rs`. `EXT_TO_MIME` and
85 `SUPPORTED_MIME_TYPES` are derived from it — do not hand-edit either. See
86 `mime-detection-routing` for the full procedure, including the count assertion to bump.
872. Create an extractor implementing `InternalDocumentExtractor` (not `DocumentExtractor`).
883. Set `supported_mime_types()` and `priority()` (default 50).
894. Register in `extractors/mod.rs → register_default_extractors()`.
905. Feature-gate if optional: `#[cfg(feature = "my-format")]`.
916. Apply `SecurityBudget` / `SecurityLimits` to any user-supplied content.
927. Add `#[cfg_attr(alef, alef(skip))]` to the extractor struct or the binding regen aborts.
938. Add tests with fixture files (see the `test-corpus` skill for where fixtures come from).