# Benchmark Htr Engines > Select an OCR/HTR engine by benchmarking candidates on the same labelled samples — source ground truth from open datasets (e.g. Hugging Face PAGE-XML corpora) or hand-transcription, wrap every engine behind a transcribe(image_path) -> str adapter (vision-LLMs via any OpenAI-compatible endpoint, dedicated HTR REST APIs, cloud OCR), and score with zero-dependency raw CER, lenient CER, WER, and a critical name/date token diff. Use when choosing an engine for handwriting or historical-document transcription, when published accuracy claims need honest verification on your own material, or when a records workflow must surface confident-but-wrong names and dates that a low aggregate CER hides. - Skill: `pjt222/benchmark-htr-engines` (Agent Skill, multi-file: 2 files) - Install (CLI): `npx skillmds@latest add pjt222/benchmark-htr-engines` - Raw SKILL.md: https://api.skillmd.com/api/skills/pjt222/benchmark-htr-engines/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: DevOps & Infra - License: MIT - Author: pjt222 (https://skillmd.com/u/pjt222) - Updated: 2026-09-10 - Page: https://skillmd.com/skills/pjt222/benchmark-htr-engines --- # Benchmark HTR Engines Choose a handwritten-text-recognition (HTR) or OCR engine with evidence instead of vendor claims: run every candidate over the same labelled samples and compare raw CER, lenient CER, WER, and — the decisive signal for records work — a critical name/date token diff. The harness is deliberately self-contained: scoring implements edit distance directly (no third-party dependency), and a fully mocked self-test validates the math with no API key and no network. ## When to Use - Selecting between vision-LLMs, dedicated HTR services (Transkribus-style REST APIs), and cloud OCR for a handwriting or historical-document task - Verifying published or vendor accuracy claims on your own material — script, language, ink, and layout rarely match the benchmark corpus behind the claim - Working with genealogy, archival, or register material where a fluent transcription containing one wrong name or year is the real failure mode - Re-ranking engines after new models ship, against a frozen sample set - Any comparison where transcription conventions (line-break hyphenation, case, ß/ss) would otherwise pollute the accuracy signal ## Inputs - **Required**: 10–20 labelled samples — image + ground-truth text pairs spanning easy to hard material (or an open dataset to pull them from, Step 2) - **Required**: Two or more candidate engines (model ids or service endpoints) - **Optional**: API keys per engine family (e.g. `LLM_API_KEY`, HTR service credentials) — the scoring self-test needs none - **Optional**: Critical-token rules (default: any token containing a digit, month names including archaic forms, Capitalised words) - **Optional**: Lenient-CER folds (default: line-break hyphenation, whitespace, case, ß→ss; adjust per language and script) ## Procedure ### Step 1: Lay Out the Sample Corpus Use a flat directory of image + ground-truth pairs sharing a stem. This layout is the whole data contract — every engine and the scorer read only this. ```text samples/ register_p034.jpg # the page or line image (.jpg .jpeg .png .tif .tiff .webp) register_p034.gt.txt # its ground-truth transcription (UTF-8) ``` Guidelines: - 10–20 line crops or a few full pages are enough to separate engines; spread them easy → hard (clean vs. faded ink, marginalia, mixed hands). - Line-level crops score more cleanly on a first pass; full pages also work — vision-LLMs read them whole, HTR services run their own line detection. - If the material is private (church, court, medical records), gitignore `samples/` immediately, before the first commit. **Expected:** Every image in `samples/` has a same-stem `.gt.txt`; private material is covered by a `.gitignore` rule from the start. **On failure:** Orphan images (no `.gt.txt`) are skipped by the runner — either transcribe them or remove them. If private samples were already committed, rewriting history is required; prevention (gitignore first) is far cheaper. ### Step 2: Source Ground Truth Two routes; both end in the Step 1 layout. **Route A — open dataset.** Pull a handful of labelled rows from a public corpus matching your script, e.g. Hugging Face `dh-unibe/image-text_kurrent-xix` (19th-century German Kurrent, MIT licence, DOI 10.57967/hf/8590). Each row is an image plus PAGE XML; parse the plain text out of the XML. Stream — never download the full corpus: ```python from pathlib import Path from datasets import load_dataset # pip install datasets dataset = load_dataset("dh-unibe/image-text_kurrent-xix", split="train", streaming=True) written = 0 for row in dataset: if written >= 12: break ground_truth = plain_text_from_page_xml(row["xml_content"]).strip() if not (10 <= len(ground_truth) <= 600): continue # skip near-empty rows and multi-column monsters Path(f"samples/{written:02d}_kurrent.png").write_bytes(row["image"]["bytes"]) Path(f"samples/{written:02d}_kurrent.gt.txt").write_text( ground_truth, encoding="utf-8") written += 1 ``` The PAGE-XML parser must prefer each `TextLine`'s line-level `TextEquiv/Unicode` and only fall back to joining `` children — see [references/EXAMPLES.md](references/EXAMPLES.md) for a correct implementation. **Route B — hand-transcribe.** Transcribe a few target pages yourself. Transcribe the original (old) spelling faithfully — do not modernise — so CER measures the engine, not your edits. This is a labelling task; see [label-training-data](../label-training-data/SKILL.md) for conventions. **Expected:** Sample pairs on disk; ground truth preserves original spelling and line breaks. Dataset attribution recorded if results will be published. **On failure:** If streaming fails, check the `datasets` version and network; reduce `--scan-limit`-style row scanning rather than downloading the corpus. If parsed ground truth contains only the first word of each line, your PAGE parser hit the word-segmentation trap (see Common Pitfalls). A near-genre dataset (line crops, related script) is a valid accuracy signal but not a genre-exact one — note the caveat in the report. ### Step 3: Define the Engine Adapter Interface Every engine hides behind one method, so the runner and scorer never know which family they are talking to: ```python from abc import ABC, abstractmethod class TranscriptionEngine(ABC): """Common interface: given an image file, return the transcribed text.""" name: str = "engine" # e.g. "llm:vendor/model-pro", "htr:51170" @abstractmethod def transcribe(self, image_path: str) -> str: """Return the plain-text transcription of the image at image_path.""" raise NotImplementedError ``` Three adapter families cover the practical field: | Family | Transport | Notes | |---|---|---| | Vision-LLM | Any OpenAI-compatible chat endpoint (OpenRouter, vendor APIs) | One code path covers many models; swap the model id | | Dedicated HTR | REST: authenticate → submit → poll → fetch PAGE XML | Asynchronous jobs; result needs XML → text extraction | | Cloud OCR | Vendor SDK or REST call | Usually synchronous; concatenate returned text blocks | **Expected:** A `base.py` with the ABC; each candidate engine is a subclass with a distinct `name` used in reports and prediction filenames. **On failure:** If an engine cannot fit `transcribe(image_path) -> str` (e.g. it returns only word boxes), do the flattening inside the adapter — never let format differences leak into the scorer. ### Step 4: Implement Adapters with Retry-on-Empty For the vision-LLM family, send the image as a base64 data URI with a hallucination-suppressing prompt (transcribe only what is legible, mark unreadable spans `[...]`, never invent names or dates), `temperature=0`, and a generous `max_tokens`. Then guard the known transient failure: ```python # Reasoning models intermittently return EMPTY content with # finish_reason="stop" — reasoning tokens consumed, no text emitted. # Without a retry, one transient empty scores as 100% CER and poisons # that engine's mean. last_finish_reason = None for _attempt in range(self.max_retries): # e.g. 3 response = self.client.chat.completions.create( model=self.model, temperature=0, # minimise fabrication max_tokens=self.max_output_tokens, # headroom: reasoning + full page messages=messages, ) choice = response.choices[0] text = (choice.message.content or "").strip() if text: return text last_finish_reason = choice.finish_reason raise RuntimeError( f"empty content after {self.max_retries} attempts " f"(last finish_reason={last_finish_reason})") ``` For dedicated HTR REST services, the adapter wraps an asynchronous job: authenticate, POST the image (raw base64), poll the process id until `FINISHED`/`FAILED`/timeout, fetch the PAGE XML result, and extract plain text. Full templates for both families are in [references/EXAMPLES.md](references/EXAMPLES.md). **Expected:** Each adapter returns non-empty text for a legible test image, or raises with a diagnostic (not an empty string). Config errors (missing keys) raise at construction time with the variable names spelled out. **On failure:** Empty after all retries — raise `max_tokens` (reasoning tokens compete with output tokens) before blaming the model. HTTP 401 mid-poll — re-authenticate once and retry rather than abandoning a submitted (paid) job. 429/5xx — retry with capped exponential backoff. ### Step 5: Implement Zero-Dependency Scoring Implement Levenshtein distance directly (two-row dynamic programming, ~15 lines) so scoring runs anywhere with no install. Derive four metrics: - **Raw CER** — character edit distance / reference length. Honest but inflated by convention differences. - **Lenient CER** — CER after folding transcription conventions, so it reflects recognition, not orthography: ```python def normalize_for_lenient_cer(text: str) -> str: normalized = unicodedata.normalize("NFC", text) normalized = LINEBREAK_HYPHEN.sub("", normalized) # join hyphenated breaks normalized = WHITESPACE.sub(" ", normalized) # layout != recognition normalized = normalized.lower() # case != recognition normalized = normalized.replace("ß", "ss") # script-specific folds return normalized.strip() ``` Keep it deliberately conservative — over-normalising hides real errors. Report lenient CER as a secondary metric next to raw CER, never instead. - **WER** — the same edit distance over whitespace-split words. - **Critical name/date token diff** — a word-level diff (`difflib.SequenceMatcher`) where each edit is flagged if it touches a critical token: ```python def is_critical_token(token: str) -> bool: stripped = token.strip(".,;:()[]{}\"'-") if not stripped: return False if any(ch.isdigit() for ch in stripped): # dates, years, ages return True if stripped.lower() in MONTH_NAMES: # incl. archaic forms like "7ber" return True # Capitalised word -> likely proper noun (over-flags German common # nouns by design: safer to surface too much for human review) return stripped[0].isalpha() and stripped[0].isupper() \ and stripped[1:] == stripped[1:].lower() ``` The critical-token count is the records-relevant signal: a confident-wrong surname or year is invisible inside a good-looking aggregate CER. **Expected:** A `scoring.py` importing only the standard library, exposing `character_error_rate`, `lenient_character_error_rate`, `word_error_rate`, and `critical_token_diff`. **On failure:** If lenient CER ever exceeds raw CER on the same pair, the normalizer is corrupting text (check regex ordering: fold hyphenated line breaks *before* collapsing whitespace). Empty-reference edge cases must return 0.0 for empty-vs-empty and 1.0 for empty-vs-nonempty, not divide by zero. ### Step 6: Validate Scoring with a No-Key Self-Test Before spending any API budget, assert the math on synthetic tokens (all names/dates below are invented): ```python assert character_error_rate("Haus", "Hans") == 0.25 # 1 edit / 4 chars assert word_error_rate("geboren den 12 Merz 1834", "geboren den 12 Merz 1835") == 0.2 # 1 word of 5 # Convention folds to zero... assert lenient_character_error_rate("Ver¬\nzeichnis", "Ver-\nzeichnis") == 0.0 assert lenient_character_error_rate("Grüßner", "Grüssner") == 0.0 # ...but a genuine name misread survives lenient folding assert lenient_character_error_rate("Beispielmann", "Beispelmann") > 0.0 edits = critical_token_diff( "getauft den 12 Merz 1834 Georg Beispielmann", "getauft den 12 Merz 1835 Georg Beispelmann") assert sum(1 for e in edits if e.touches_critical) == 2 # year AND surname print("scoring self-test: PASS") ``` Wire it into the runner as `--selftest` so it is always one flag away. If an adapter has network seams (the REST family), mock them in the self-test too — the full template in [references/EXAMPLES.md](references/EXAMPLES.md) runs without `requests` even installed. **Expected:** `python3 run_benchmark.py --selftest` prints `PASS` with no API key, no network, and no third-party package. **On failure:** A failing assertion is a scoring bug — fix it before running engines, or every downstream comparison is untrustworthy. Never weaken an assertion to make it pass; re-derive the expected value by hand first. ### Step 7: Run the Panel and Report Run all candidate engines over all samples in one invocation. Per engine × sample: score, print the per-sample line plus only the critical edits, and save the raw prediction (re-scoring later must not require re-paying for API calls). Keep one engine's failure from killing the panel — catch per-sample errors, count them, continue. ```text 07_kurrent.png CER=6.2% lenient-CER=3.1% WER=11.0% name/date-errors=1 !! NAME/DATE [replace] gt: '1834' -> pred: '1835' (+ 2 minor non-critical edit(s)) ===== COMPARISON (sorted by lenient CER, lower is better) ===== engine CER lenient name/date ok llm:vendor/model-pro 7.9% 4.1% 3 12/12 htr:rest-model-51170 11.2% 6.0% 5 12/12 llm:vendor/model-flash 13.5% 9.8% 14 11/12 ``` Persist everything: the full console report (e.g. `reports/last_run.txt`) and one `reports/..pred.txt` per prediction. **Expected:** A comparison table sorted by lenient CER; per-sample critical diffs; raw predictions on disk for every successful engine × sample pair. **On failure:** An engine erroring on every sample is a config problem (key, model id, endpoint) — its constructor should have said so; check the first error message, not the aggregate. If two engines tie on lenient CER, rank by critical-token count next. ### Step 8: Interpret and Decide Read the table in this priority order: 1. **Critical name/date errors** — the failure that corrupts records; a fluent engine with more name errors loses to a rougher one with fewer. 2. **Lenient CER** — recognition quality with convention noise removed. 3. **Raw CER / WER** — tie-breakers and sanity checks (a large raw-vs-lenient gap means convention differences, not recognition differences). Then weigh the non-accuracy dimensions — cost per page, privacy (does the image leave your infrastructure?), throughput, and licensing — covered in the companion guide [choosing-an-htr-ocr-engine](../../guides/choosing-an-htr-ocr-engine.md). **Expected:** A documented engine choice citing the table, the critical-token counts, and the non-accuracy constraints that applied. **On failure:** If no engine is acceptable (critical errors on most samples), re-scope: better images (higher DPI, line crops), a fine-tuned model for the script, or human transcription with engine assist rather than engine-first. ## Validation - [ ] Every image in `samples/` has a same-stem `.gt.txt` (no orphans scored) - [ ] `run_benchmark.py --selftest` passes with no API key, network, or third-party install - [ ] Every LLM adapter retries on empty content before raising - [ ] Report shows raw CER, lenient CER, WER, and critical name/date count per engine - [ ] Comparison table is sorted by lenient CER and states samples-ok per engine - [ ] Raw predictions saved to disk for every successful engine × sample - [ ] Private sample text appears nowhere outside `samples/` (gitignored); docs and tests use invented tokens only - [ ] Dataset licence and attribution recorded if results will be published ## Common Pitfalls - **Trusting aggregate CER**: A 4% CER engine that swaps one surname per page is worse for records work than an 8% engine that gets names right. Always read the critical-token diff before the means. - **Over-normalising lenient CER**: Folding too much (punctuation, diacritics, word order) hides real recognition errors. Fold only documented transcription conventions, and always report raw CER alongside. - **Empty LLM responses scored as 100% CER**: Reasoning models intermittently return empty content with `finish_reason=stop`. Without retry-on-empty, one transient blip destroys that engine's mean and your ranking. - **PAGE-XML first-word trap**: Word-segmented PAGE XML serializes `` children before the line-level ``; a naive pre-order `iter()` walk returns only the first word of every line — ground truth silently truncates and every engine looks terrible. Prefer the line-level `TextEquiv`. - **Modernised ground truth**: "Correcting" old spelling in the ground truth charges every engine for errors it did not make. Transcribe faithfully; handle convention differences in the lenient normalizer instead. - **Committing private samples**: Church, court, or family records must be gitignored before the first commit, and never pasted into prompts, tests, or documentation — use invented tokens (e.g. `Beispielmann, Georg *1834`). - **One-sample verdicts**: A single easy sample cannot separate engines. Use 10–20 samples spanning difficulty, and report per-sample rows so one outlier is visible instead of buried in a mean. ## Related Skills - [label-training-data](../label-training-data/SKILL.md) - hand-transcribing ground truth is a labelling task; reuse its conventions and QA loop - [run-ab-test-models](../run-ab-test-models/SKILL.md) - statistical comparison when two shortlisted engines are close and the sample budget can grow - [manage-token-budget](../manage-token-budget/SKILL.md) - controlling vision-LLM spend when the panel or sample set grows - [choosing-an-htr-ocr-engine](../../guides/choosing-an-htr-ocr-engine.md) - companion guide: the engine landscape, cost, privacy, and decision criteria this benchmark feeds See [references/EXAMPLES.md](references/EXAMPLES.md) for the complete generalized harness template (scoring, adapters, runner, self-test, dataset fetcher).