890 Code Adaptation Ab978a8a

Code Adaptation Guide

When and how to modify ADAPTABLE code when configuration isn't enough.

When to Modify Code

Only modify code when:

1. Configuration changes have been tried (2-3 attempts)
2. The problem is clearly in heuristics, not data
3. The required change is isolated to ADAPTABLE files

Never modify FROZEN files (see architecture.md).

ADAPTABLE Files Reference

Pattern 1: Adding a New Strategy

When the default SimpleStrategy doesn't work for a specific PDF type.

Steps

1. Create new file in strategies/:

# strategies/academic_strategy.py

from .base_strategy import BaseStrategy
from detectors.reading_order.xy_cut_sorter import XYCutSorter

class AcademicStrategy(BaseStrategy):
    """Strategy for academic papers with columns and citations."""

    def order_blocks(self, blocks, config):
        # Always use XY-cut for academic papers
        sorter = XYCutSorter()
        ordered, confidence = sorter.sort(blocks)
        return ordered, confidence

    def detect_structure(self, blocks, config, images, metadata):
        # Custom structure detection for academic format
        # ... your implementation
        pass

2. Register in converter.py:

from .strategies.academic_strategy import AcademicStrategy

STRATEGY_REGISTRY = {
    "simple": SimpleStrategy,
    "academic": AcademicStrategy,  # Add this
}

3. Use the new strategy:

converter = Converter(strategy="academic")
result = converter.convert(pdf_path, epub_path, config)

Pattern 2: Adding a Footnote Pattern

When footnotes use a non-standard format.

Steps

1. Identify the pattern in the PDF:

import fitz
doc = fitz.open("input.pdf")
text = doc[5].get_text()
# Look for: †1, *1, ^1, etc.

2. Add pattern to footnote_detector.py:

# In detectors/footnote_detector.py

PATTERNS = {
    'bracket': re.compile(r'\[(\d{1,2})\]'),
    'paren': re.compile(r'\((\d{1,2})\)'),
    'period': re.compile(r'(?<=\w)\.(\d{1,2})(?=\s|$)'),
    'superscript': re.compile(r'(?<=[a-zA-Z])(\d{1,2})(?=\s|$)'),
    # Add your pattern:
    'dagger': re.compile(r'†(\d{1,2})'),
    'asterisk': re.compile(r'\*(\d{1,2})'),
}

3. Test:

from conversion.detectors.footnote_detector import FootnoteDetector

detector = FootnoteDetector(patterns=['dagger'])
refs = detector.find_references("This statement†1 is cited.")
print(refs)  # [FootnoteRef(number=1, original_text='†1', ...)]

Pattern 3: Tuning Heading Detection

When the default heuristics don't identify headings correctly.

Steps

1. Understand the current scoring in structure_classifier.py:

def _calculate_heading_score(self, block, prev, next_b, body_size, body_flags):
    score = 0.0
    signals = []

    # Font size check
    if block.font_size > body_size * self.heading_threshold:
        score += 0.4
        signals.append(f"font_size +0.4")

    # Bold check
    if block.flags & 16:  # Bold flag
        score += 0.2
        signals.append("bold +0.2")

    # Short text (headings are usually short)
    if len(block.text) < 100:
        score += 0.1
        signals.append("short +0.1")

    # ... more signals
    return score, signals

2. Add new signals or adjust weights:

def _calculate_heading_score(self, block, prev, next_b, body_size, body_flags):
    score = 0.0
    signals = []

    # Existing checks...

    # NEW: Check for all-caps (common heading style)
    if block.text.isupper() and len(block.text) < 50:
        score += 0.3
        signals.append("all_caps +0.3")

    # NEW: Check for centered text (approximate)
    page_center = 0.5  # Assume page center
    block_center = (block.x0 + block.x1) / 2 / page_width
    if abs(block_center - page_center) < 0.1:
        score += 0.15
        signals.append("centered +0.15")

    return score, signals

3. Test with debug output:

classifier = StructureClassifier()
blocks = classifier.classify(text_blocks)

for block in blocks:
    if block.score > 0:
        print(f"{block.role}: {block.original_block.text[:50]}")
        print(f"  Score: {block.score}, Signals: {block.debug_signals}")

Pattern 4: Custom Reading Order

When neither y_sort nor xy_cut produces correct order.

Steps

1. Create new sorter in reading_order/:

# detectors/reading_order/custom_sorter.py

from .base import BlockSorter

class CustomSorter(BlockSorter):
    """Custom reading order for specific layout type."""

    def sort(self, blocks):
        # Your custom sorting logic
        # Example: Sort by sections first, then by position within section

        sections = self._detect_sections(blocks)
        ordered = []

        for section in sections:
            # Sort blocks within section by y, then x
            section_blocks = sorted(
                [b for b in blocks if self._in_section(b, section)],
                key=lambda b: (b.y0, b.x0)
            )
            ordered.extend(section_blocks)

        confidence = self._calculate_confidence(ordered)
        return ordered, confidence

    def _detect_sections(self, blocks):
        # Detect logical sections (e.g., by horizontal lines)
        ...

    def _in_section(self, block, section):
        # Check if block belongs to section
        ...

    def _calculate_confidence(self, ordered):
        # Estimate confidence in ordering
        return 0.85

2. Register sorter:

# In simple_strategy.py or base_strategy.py
from .detectors.reading_order.custom_sorter import CustomSorter

def order_blocks(self, blocks, config):
    if config.reading_order_strategy == "custom":
        sorter = CustomSorter()
    elif config.reading_order_strategy == "xy_cut":
        sorter = XYCutSorter()
    else:
        sorter = YSorter()

    return sorter.sort(blocks)

Pattern 5: Modifying HTML Output

When EPUB styling needs customization.

EndnoteFormatter Customization

# In detectors/endnote_formatter.py

class EndnoteFormatter:
    def format_endnotes(self, blocks):
        html_parts = []

        for block in blocks:
            if block.role != "endnote":
                continue

            num = block.endnote_num
            content = self._extract_content(block)

            # Customize the HTML output:
            html_parts.append(
                f'<div class="endnote" id="note{num}">'
                f'  <span class="endnote-number">{num}</span>'
                f'  <span class="endnote-text">{content}</span>'
                f'</div>'
            )

        return '\n'.join(html_parts)

EPUBBuilder Stylesheet

To modify default styles, edit core/epub_builder.py:

def _write_stylesheet(self, temp_path):
    css = '''
    /* Custom styles */
    .endnote {
        margin: 1em 0;
        padding-left: 2em;
        text-indent: -2em;
    }

    .endnote-number {
        font-weight: bold;
        color: #0066cc;
    }
    '''
    # ... write to file

Note: epub_builder.py is technically FROZEN, but stylesheet changes are cosmetic and safe.

Testing Your Changes

After any code modification:

1. Run unit tests:

python -m pytest tests/ -v

2. Run specific test file:

python -m pytest tests/conversion/test_structure_classifier.py -v

3. Test on real PDF:

python -m scripts.convert input.pdf output.epub
python -m scripts.validate input.pdf output.epub

4. Verify validation passes:

Completeness: > 95%
Order score: > 80%

Documenting Changes

When you modify code:

1. Add comment explaining the change:

# MODIFIED: Added all-caps heading detection for Book XYZ
# Original threshold was insufficient for this PDF style
if block.text.isupper():
    score += 0.3

2. Update tests if behavior changes
3. Consider if change should be configurable instead

Rollback Strategy

If changes break things:

1. Git revert:

git checkout -- path/to/file.py

2. Or restore from known good state:

git stash
python -m pytest tests/  # Verify tests pass
git stash pop  # Re-apply changes

When to Ask for Help

If you're unsure about a change:

• It affects multiple files
• It changes fundamental algorithms
• Tests start failing
• You need to modify FROZEN files

Stop and reconsider the approach. Often there's a configuration solution that was missed.