Feature Requirements: Structure Analyzer

Feature Requirements: Structure Analyzer

Problem Statement

PDF files are essentially "flat" collections of text blocks with position coordinates. They lack semantic structure such as Chapters, Sections, Paragraphs, Lists, and Footnotes. To create a high-quality, navigable EPUB, we must deduce this structure from visual cues (font size, bold weight, spacing) and textual patterns. Without this, the EPUB would be a single continuous stream of text without a Table of Contents or proper formatting.

Goals & Objectives

• Automated Structure Detection: Identify high-level structure (Chapters, Sub-chapters) to generate a Table of Contents (ToC).
• Semantic Tagging: Classify text blocks into semantic roles: Paragraph, Heading (H1-H3), List Item, Quote, Footnote.
• Footnote Linking: Associate footnote markers in the text (e.g., [1], *) with their corresponding content at the bottom of the page/chapter.
• Configurable Heuristics: Allow users to tweak detection rules (e.g., "Headers must be >14pt bold") via configuration.

Non-Goals

• OCR: We assume the PDF has a text layer (or is pre-processed).
• Complex Layouts (v1): We focus on single/double-column text. Complex magazine-style layouts with floating boxes are best-effort.
• Image Analysis: We do not analyze images for content, only their position.

User Stories

1. Chapter Detection: "As a user, I want chapter titles to be automatically recognized so that my EPUB has a working Table of Contents."
2. Hierarchy Preservation: "As a reader, I want to see distinct formatting for main headings vs sub-headings to understand the book's structure."
3. Footnote Navigation: "As a scholar, I want to click on a footnote number and jump to the definition, then jump back, instead of manually scrolling."
4. Noise Filtering: "As a user, I want the system to automatically identify and remove repetitive headers/footers (e.g., 'Page X of Y', 'My Book Title') so they don't interrupt the reading flow."

Success Criteria

• Header Recall: >90% of explicit chapter headers detected correctly in standard fiction/non-fiction layouts.
• ToC Generation: Generated Table of Contents matches the visual structure of the PDF.
• Footnote Linking: >80% of standard numerical footnotes (1, [1]) are correctly linked to their definitions.
• Performance: Analysis of a 300-page book takes < 30 seconds.

Constraints & Assumptions

• Input: List of TextBlock objects from PDFExtractor (with font/position data).
• Heuristics: Detection relies on the consistency of the PDF styling (e.g., all H1s use the same font).
• Scope: Footnotes may range from "bottom of page" to "end of chapter".

Data Models (Draft)

@dataclass
class StructuredBlock:
    text: str
    role: str  # "p", "h1", "h2", "li", "footnote", "quote"
    level: int = 0
    id: str = None  # ToC anchor or Footnote ID

Questions & Open Items

• How do we handle "Run-in heads" (bold text at start of paragraph)?
• Handling multi-line headers? (Merge logic needed).
• Detecting "drop caps" (large first letter)?