Feature Requirements: Structure Analyzer
Problem Statement
PDF files are essentially "flat" collections of text blocks with position coordinates. They lack semantic structure such as Chapters, Sections, Paragraphs, Lists, and Footnotes. To create a high-quality, navigable EPUB, we must deduce this structure from visual cues (font size, bold weight, spacing) and textual patterns. Without this, the EPUB would be a single continuous stream of text without a Table of Contents or proper formatting.
Goals & Objectives
- Automated Structure Detection: Identify high-level structure (Chapters, Sub-chapters) to generate a Table of Contents (ToC).
- Semantic Tagging: Classify text blocks into semantic roles:
Paragraph, Heading (H1-H3), List Item, Quote, Footnote.
- Footnote Linking: Associate footnote markers in the text (e.g.,
[1], *) with their corresponding content at the bottom of the page/chapter.
- Configurable Heuristics: Allow users to tweak detection rules (e.g., "Headers must be >14pt bold") via configuration.
Non-Goals
- OCR: We assume the PDF has a text layer (or is pre-processed).
- Complex Layouts (v1): We focus on single/double-column text. Complex magazine-style layouts with floating boxes are best-effort.
- Image Analysis: We do not analyze images for content, only their position.
User Stories
- Chapter Detection: "As a user, I want chapter titles to be automatically recognized so that my EPUB has a working Table of Contents."
- Hierarchy Preservation: "As a reader, I want to see distinct formatting for main headings vs sub-headings to understand the book's structure."
- Footnote Navigation: "As a scholar, I want to click on a footnote number and jump to the definition, then jump back, instead of manually scrolling."
- Noise Filtering: "As a user, I want the system to automatically identify and remove repetitive headers/footers (e.g., 'Page X of Y', 'My Book Title') so they don't interrupt the reading flow."
Success Criteria
- Header Recall: >90% of explicit chapter headers detected correctly in standard fiction/non-fiction layouts.
- ToC Generation: Generated Table of Contents matches the visual structure of the PDF.
- Footnote Linking: >80% of standard numerical footnotes (
1, [1]) are correctly linked to their definitions.
- Performance: Analysis of a 300-page book takes < 30 seconds.
Constraints & Assumptions
- Input: List of
TextBlock objects from PDFExtractor (with font/position data).
- Heuristics: Detection relies on the consistency of the PDF styling (e.g., all H1s use the same font).
- Scope: Footnotes may range from "bottom of page" to "end of chapter".
Data Models (Draft)
@dataclass
class StructuredBlock:
text: str
role: str # "p", "h1", "h2", "li", "footnote", "quote"
level: int = 0
id: str = None # ToC anchor or Footnote ID
Questions & Open Items
- How do we handle "Run-in heads" (bold text at start of paragraph)?
- Handling multi-line headers? (Merge logic needed).
- Detecting "drop caps" (large first letter)?
1---2name: 907-feature-structure-analyzer-51a3dc463description: Requirements for the PDF structure analysis and content detection module.4---56# Feature Requirements: Structure Analyzer78## Problem Statement9PDF files are essentially "flat" collections of text blocks with position coordinates. They lack semantic structure such as Chapters, Sections, Paragraphs, Lists, and Footnotes. To create a high-quality, navigable EPUB, we must deduce this structure from visual cues (font size, bold weight, spacing) and textual patterns. Without this, the EPUB would be a single continuous stream of text without a Table of Contents or proper formatting.1011## Goals & Objectives12- **Automated Structure Detection**: Identify high-level structure (Chapters, Sub-chapters) to generate a Table of Contents (ToC).13- **Semantic Tagging**: Classify text blocks into semantic roles: `Paragraph`, `Heading (H1-H3)`, `List Item`, `Quote`, `Footnote`.14- **Footnote Linking**: Associate footnote markers in the text (e.g., `[1]`, `*`) with their corresponding content at the bottom of the page/chapter.15- **Configurable Heuristics**: Allow users to tweak detection rules (e.g., "Headers must be >14pt bold") via configuration.1617### Non-Goals18- **OCR**: We assume the PDF has a text layer (or is pre-processed).19- **Complex Layouts (v1)**: We focus on single/double-column text. Complex magazine-style layouts with floating boxes are best-effort.20- **Image Analysis**: We do not analyze images for content, only their position.2122## User Stories231. **Chapter Detection**: "As a user, I want chapter titles to be automatically recognized so that my EPUB has a working Table of Contents."242. **Hierarchy Preservation**: "As a reader, I want to see distinct formatting for main headings vs sub-headings to understand the book's structure."253. **Footnote Navigation**: "As a scholar, I want to click on a footnote number and jump to the definition, then jump back, instead of manually scrolling."264. **Noise Filtering**: "As a user, I want the system to automatically identify and remove repetitive headers/footers (e.g., 'Page X of Y', 'My Book Title') so they don't interrupt the reading flow."2728## Success Criteria29- **Header Recall**: >90% of explicit chapter headers detected correctly in standard fiction/non-fiction layouts.30- **ToC Generation**: Generated Table of Contents matches the visual structure of the PDF.31- **Footnote Linking**: >80% of standard numerical footnotes (`1`, `[1]`) are correctly linked to their definitions.32- **Performance**: Analysis of a 300-page book takes < 30 seconds.3334## Constraints & Assumptions35- **Input**: List of `TextBlock` objects from `PDFExtractor` (with font/position data).36- **Heuristics**: Detection relies on the consistency of the PDF styling (e.g., all H1s use the same font).37- **Scope**: Footnotes may range from "bottom of page" to "end of chapter".3839## Data Models (Draft)40```python41@dataclass42class StructuredBlock:43 text: str44 role: str # "p", "h1", "h2", "li", "footnote", "quote"45 level: int = 046 id: str = None # ToC anchor or Footnote ID47```4849## Questions & Open Items50- How do we handle "Run-in heads" (bold text at start of paragraph)?51- Handling multi-line headers? (Merge logic needed).52- Detecting "drop caps" (large first letter)?
tools-only