# 897 Feature Text Canonicalizer 239b7661

> Task breakdown for implementing the text canonicalizer.

- Skill: `tools-only/897-feature-text-canonicalizer-239b7661` (Agent Skill, multi-file: 3 files)
- Install (CLI): `npx skillmds@latest add tools-only/897-feature-text-canonicalizer-239b7661`
- Raw SKILL.md: https://api.skillmd.com/api/skills/tools-only/897-feature-text-canonicalizer-239b7661/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Productivity
- Author: tools-only (https://skillmd.com/u/tools-only)
- Updated: 2026-09-22
- Page: https://skillmd.com/skills/tools-only/897-feature-text-canonicalizer-239b7661

---


# Project Planning & Task Breakdown

## Milestones
**What are the major checkpoints?**

- [x] Milestone 1: Basic Unicode and ligature resolution.
- [x] Milestone 2: Hyphenation cleanup and aggressive mode.
- [x] Milestone 3: Full test coverage.

## Task Breakdown
**What specific work needs to be done?**

### Phase 1: Foundation
- [ ] Task 1.1: Implement `resolve_ligatures` with a mapping of common PDF ligatures.
- [ ] Task 1.2: Implement `remove_hyphenation` (handling `-\n` scenarios).
- [ ] Task 1.3: Wrap them into a main `canonicalize` function using `unicodedata.nfkc`.

### Phase 2: Refinement
- [ ] Task 2.1: Add "aggressive" mode for OCR artifacts (optional common fixes).
- [ ] Task 2.2: Character-level logging for debugging canonicalization gaps.

### Phase 3: Testing
- [ ] Task 3.1: Write unit tests covering:
    - [ ] Unicode normalization (different forms).
    - [ ] Ligatures (ff, fi, fl, ffi, ffl, ae, oe).
    - [ ] Hyphenation across lines.
    - [ ] Combination of all effects.

## Dependencies
**What needs to happen in what order?**

- Safe to build in parallel with `text_segmenter`, but `segmenter` will eventually receive canonicalized text.

## Timeline & Estimates
**When will things be done?**

- Implementation: 1.5 hours
- Testing: 1 hour

## Risks & Mitigation
**What could go wrong?**

- **Risk:** Removing a hyphen that was actually part of a word (e.g. "semi-detached").
- **Mitigation:** Only remove hyphens followed by whitespace/newlines.