1767 Feature PDF Extractor Ef202632

Project Planning & Task Breakdown

Milestones

What are the major checkpoints?

• Milestone 1: Library integration and basic extraction (Done).
• Milestone 2: Paragraph reconstruction and canonicalization (Done).
• Milestone 3: Unit tests and integration with real fixtures (Done).
• Milestone 4: Smart Noise Removal (Headers/Footers) (Done).

Task Breakdown

What specific work needs to be done?

Phase 1: Foundation

• Task 1.1: Install pymupdf in .venv. (Done)
• Task 1.2: Implement PDFExtractor class skeleton with resource management (context manager).
• Task 1.3: Implement text extraction using page.get_text("blocks").
• Task 1.4: Add iter_pages for memory-efficient processing.

Phase 2: Refinement

• Task 2.1: Implement smart block joining logic for paragraphs.
• Task 2.2: Integrate canonicalize from validation.text_canonicalizer.
• Task 2.3: Extract metadata and handle encryption/corruption errors.

Phase 3: Advanced Extraction (Added during Phase 3 Validation)

• Task 3.1: Implement _identify_noise_patterns using statistical analysis of repeated lines.
• Task 3.2: Implement regex-based stripping of page numbers and copyrights.
• Task 3.3: Strip noise patterns from within text blocks to handle embedded artifacts.

Phase 4: Testing

• Task 4.1: Unit tests for basic extraction logic.
• Task 4.2: Integration test showing 100% text recovery after noise removal.

Timeline & Estimates

• Completed 2025-12-30.

Risks & Mitigation

• Risk: Poor reading order in multi-column PDFs.
• Mitigation: Used page.get_text("blocks", sort=True) to respect layout blocks.
• Risk: Recurrent noise (headers/footers) breaking validation.
• Mitigation: Implemented heuristic-based identification and removal of recurring lines.