896 Feature PDF To Epub Conversion 4078e23f

Implementation Plan

Overview

This plan covers Phase 5 of the Product Vision: implementing the conversion pipeline that transforms extracted PDF structure into valid EPUB files.

Prerequisites

Completed Dependencies:

• ✅ Phase 1-4: PDFExtractor, TextSegmenter, reading order sorters, structure detectors
• ✅ Validation components: CompletenessChecker, OrderChecker, TextCanonicalizer
• ✅ Test fixtures and integration test framework

Required Before Start:

• Review Product Vision Phase 5 section
• Review design document (feature-pdf-to-epub-conversion.md)
• Verify all existing tests pass (56/56)

Task Breakdown

Phase 1: Foundation (2-3 hours)

Task 1.1: Implement BaseStrategy

File: pdf_to_epub/conversion/strategies/base_strategy.py Description: Create abstract base class defining strategy interface Details:

• Define template method convert(pdf_path, config) -> StructuredContent
• Add hook methods: extract(), order_blocks(), detect_structure()
• Add utility method: _validate_config(config)
• Include comprehensive docstrings

Acceptance:

• BaseStrategy is abstract (cannot be instantiated)
• All hook methods raise NotImplementedError
• Type hints for all parameters and return values

Status: ✅ Completed - Dec 30, 2025

Task 1.2: Create StructuredContent data models

File: pdf_to_epub/conversion/models.py (new) Description: Define data classes for structured content representation Details:

• StructuredContent (chapters, metadata, confidence)
• Chapter (title, level, content, footnotes)
• Footnote (marker, id, text)
• ConversionResult (epub_path, status, confidence, log)
• ConversionLog (timestamp, strategy, config, steps, warnings, errors)

Acceptance:

• All models use @dataclass decorator
• Include from_dict() and to_dict() methods
• Add validation in __post_init__() if needed

Status: ✅ Completed - Dec 30, 2025 (12 dataclasses created)

Phase 2: Simple Strategy (4-5 hours)

Task 2.1: Implement SimpleStrategy.extract()

File: pdf_to_epub/conversion/strategies/simple_strategy.py (new) Description: Extract text blocks, images, and metadata from PDF Details:

• Call PDFExtractor with config parameters
• Apply exclude_regions from config
• Extract images using PyMuPDF image extraction
• Extract metadata from PDF info dict (title, author, subject, keywords, etc.)
• Return list of TextBlock objects, images, and metadata

Acceptance:

• Uses existing PDFExtractor from core/
• Applies top/bottom/left/right exclusions correctly
• Extracts images in PNG/JPEG/GIF formats
• Reads metadata from PDF info (fallback to "Unknown" if missing)
• Returns blocks with bbox, text, font info

Status: ✅ Completed - Dec 30, 2025

Task 2.2: Implement SimpleStrategy.order_blocks()

File: Same as 2.1 Description: Sort blocks into reading order Details:

• Use YSorter from detectors/reading_order/
• Return ordered blocks + confidence score
• Confidence = 1.0 for Y-sort (always deterministic)

Acceptance:

• Blocks are sorted top-to-bottom
• Returns tuple: (ordered_blocks, 1.0)
• No blocks are dropped or duplicated

Status: ✅ Completed - Dec 30, 2025

Task 2.3: Implement SimpleStrategy.detect_structure()

File: Same as 2.1 Description: Classify blocks and build chapter structure with metadata Details:

• Use FontAnalyzer to identify heading fonts
• Use StructureClassifier to mark headings vs paragraphs
• Use StructureBuilder to group into chapters
• Create StructuredContent with chapters, metadata, and images

Acceptance:

• Chapters have titles extracted from H1 blocks
• Paragraphs are concatenated as HTML <p> tags
• Empty chapters are handled (e.g., books without chapter markers)
• Metadata (title, author, language) included in StructuredContent
• Images list included in StructuredContent
• Returns valid StructuredContent object

Status: ✅ Completed - Dec 30, 2025

Phase 3: EPUB Builder (5-6 hours)

Task 3.1: Implement EPUBBuilder.build() - Core Structure

File: pdf_to_epub/core/epub_builder.py Description: Create EPUB3 directory structure and package with metadata Details:

• Create temp directory with EPUB structure
• Write mimetype file (no compression)
• Write META-INF/container.xml
• Write OEBPS/content.opf with metadata (title, author, language, ISBN) and spine

Acceptance:

• mimetype is first file, uncompressed, contains "application/epub+zip"
• container.xml points to OEBPS/content.opf
• content.opf includes all XHTML files in spine
• content.opf includes extracted metadata (title, author, language, publisher, ISBN)

Status: ✅ Completed - Dec 30, 2025

Task 3.2: Implement EPUBBuilder - Content and Images

File: Same as 3.1 Description: Generate XHTML chapter files and embed images Details:

• Create chapterN.xhtml for each chapter
• Use XHTML5 template with proper DOCTYPE
• Include CSS stylesheet link
• Escape HTML entities in content
• Save images to OEBPS/images/ folder
• Add <img> tags in XHTML where images appear
• Update content.opf manifest with image items

Acceptance:

• XHTML files are well-formed (validate with lxml)
• Chapter titles are in <h1> tags
• Paragraphs are in <p> tags
• Special characters are escaped (e.g., &, <, >)
• Images saved as OEBPS/images/img-page-N-M.{png|jpg|gif}
• XHTML includes <img src="images/img-page-N-M.png"/> tags
• content.opf manifest includes all image items

Status: ✅ Completed - Dec 30, 2025

Task 3.3: Implement EPUBBuilder - TOC and Packaging

File: Same as 3.1 Description: Create navigation and ZIP the EPUB Details:

• Generate toc.ncx with chapter hierarchy
• Create basic stylesheet.css
• ZIP all files maintaining structure
• Rename to .epub extension

Acceptance:

• TOC includes all chapters with correct titles
• ZIP uses appropriate compression (store for mimetype, deflate for rest)
• Final file has .epub extension
• File validates with epubcheck (manual check) - Pending testing

Status: ✅ Completed - Dec 30, 2025 (epubcheck validation pending)

Phase 4: Converter Orchestrator (3-4 hours)

Task 4.1: Implement Converter.convert() - Core Logic

File: pdf_to_epub/conversion/converter.py Description: Orchestrate conversion workflow Details:

• Load config from JSON or use defaults
• Select strategy by name ("simple", "academic", etc.)
• Call strategy.convert()
• Call epub_builder.build()
• Generate ConversionLog

Acceptance:

• Handles missing config gracefully (use defaults)
• Raises clear error for unknown strategy name
• Returns ConversionResult with all fields populated

Status: ✅ Completed - Dec 30, 2025

Task 4.2: Implement Converter - Error Handling

File: Same as 4.1 Description: Handle errors and generate logs Details:

• Catch exceptions at each step
• Log errors with context (step name, file path)
• Mark ConversionResult.status as "failed" on error
• Always write log, even on failure

Acceptance:

• Encrypted PDFs fail with clear message
• Missing files fail with path in error
• Logs include timestamp, strategy, config
• Failed conversions still return ConversionResult (not exception)

Status: ✅ Completed - Dec 30, 2025

Task 4.3: Implement Config Loading and Validation

File: pdf_to_epub/conversion/converter.py Description: Load and validate conversion config (fail-fast) Details:

• Load from JSON file if provided
• Merge with default config
• Validate required fields immediately (metadata.title, metadata.author)
• Validate types (exclude_regions 0.0-1.0, page_ranges are ints)
• Validate enums (reading_order_strategy in allowed values)
• Raise ValueError with specific field name if invalid

Acceptance:

• Invalid JSON raises clear error
• Missing optional fields use defaults
• Config validation checks types and ranges before conversion starts
• Validation errors specify which field is invalid

Status: ✅ Completed - Dec 30, 2025

Phase 5: CLI Integration (2-3 hours)

Task 5.1: Implement analyze.py CLI

File: pdf_to_epub/scripts/analyze.py Description: CLI to analyze PDF structure without converting Details:

• Parse arguments: --pdf, --config, --output
• Run extraction + structure detection
• Output structured JSON report

Acceptance:

• python -m pdf_to_epub.scripts.analyze --pdf book.pdf works
• Output includes chapter titles, block count, confidence
• Handles errors gracefully

Status: ✅ Completed - Dec 30, 2025

Task 5.2: Implement convert.py CLI

File: pdf_to_epub/scripts/convert.py Description: CLI to convert PDF to EPUB Details:

• Parse arguments: --pdf, --output, --config, --strategy
• Call Converter.convert()
• Print progress and result

Acceptance:

• python -m pdf_to_epub.scripts.convert --pdf book.pdf --output book.epub works
• Shows progress messages (extracting, ordering, detecting, building)
• Prints conversion log on completion

Status: ✅ Completed - Dec 30, 2025

Task 5.3: Implement validate.py CLI

File: pdf_to_epub/scripts/validate.py Description: CLI to validate EPUB against PDF Details:

• Parse arguments: --pdf, --epub, --config
• Run completeness + order checks
• Print validation report

Acceptance:

• Uses existing CompletenessChecker and OrderChecker
• Prints pass/fail for each check
• Returns exit code 0 for pass, 1 for fail

Status: ✅ Completed - Dec 30, 2025

Dependencies

External Dependencies

• PyMuPDF (fitz) - already installed
• lxml - already installed
• None new required

Internal Dependencies

SimpleStrategy depends on:
  ├─ PDFExtractor (core/)
  ├─ YSorter (detectors/reading_order/)
  ├─ FontAnalyzer (detectors/)
  ├─ StructureClassifier (detectors/)
  └─ StructureBuilder (detectors/)

EPUBBuilder depends on:
  ├─ lxml (for XML generation)
  └─ zipfile (for packaging)

Converter depends on:
  ├─ BaseStrategy (strategies/)
  ├─ SimpleStrategy (strategies/)
  └─ EPUBBuilder (core/)

Scripts depend on:
  ├─ Converter (conversion/)
  ├─ CompletenessChecker (validation/)
  └─ OrderChecker (validation/)

Testing Strategy

Unit Tests

• test_base_strategy.py - Test abstract interface
• test_simple_strategy.py - Test each method independently
• test_epub_builder.py - Test EPUB structure and packaging
• test_converter.py - Test orchestration and error handling

Integration Tests

• test_end_to_end_conversion.py - PDF → EPUB with fixture
• test_validation_integration.py - PDF → EPUB → Validate

Test Coverage Target

• Minimum: 80% line coverage
• Critical paths: 100% coverage (error handling, config validation)

Milestones

M1: Foundation Complete ✅ (Dec 30, 2025)

• BaseStrategy implemented
• Data models defined (12 dataclasses)
• Unit tests passing - Pending

M2: Simple Strategy Complete ✅ (Dec 30, 2025)

• SimpleStrategy implemented
• Integration test with fixture passes - Pending
• Can convert simple PDF to structured content

M3: EPUB Builder Complete ✅ (Dec 30, 2025)

• EPUBBuilder creates valid EPUB3
• Manual epubcheck validation passes - Pending
• Can open EPUB in reader - Pending

M4: Full Pipeline Complete ✅ (Dec 30, 2025)

• Converter orchestrates full workflow
• CLI scripts functional (convert, analyze, validate)
• All tests passing (target: 80+ tests) - Next Phase

Risks & Mitigation

Risk 1: EPUB Validation Failures

Likelihood: High Impact: Medium Mitigation: Test with epubcheck early and often, use reference EPUB as template

Risk 2: Reading Order Errors

Likelihood: Medium Impact: High Mitigation: Reuse proven YSorter, add reading order confidence to result, validate with OrderChecker

Risk 3: Character Encoding Issues

Likelihood: Medium Impact: Low Mitigation: Always use UTF-8, escape HTML entities, test with non-ASCII text

Open Questions

• Should EPUBBuilder support EPUB2 or only EPUB3?
  • Decision: EPUB3 only - implemented in EPUBBuilder
• How to handle images in PDF?
  • Decision: Implemented image extraction using PyMuPDF, saved to OEBPS/images/
• Should conversion log be JSON or text?
  • Decision: Structured ConversionLog dataclass, CLI outputs human-readable format