Historical Ocr Eval

historical-ocr-eval

Evaluating LLMs for Historical Document OCR: A Methodological Framework for Digital Humanities — Levchenko (2025) (arXiv:2510.06743, 2025)

What this evaluates

Evaluates LLMs' ability to accurately transcribe historical 18th-century Russian documents while preserving period-specific orthography and avoiding anachronistic character insertions. It probes both standard OCR accuracy and historical fidelity under varying input contexts and prompt strategies.

Datasets

• 18th-century Russian Civil Font Texts — total ?; splits: test (-1); repo https://github.com/mary-lev/historical-ocr-analysis

Metrics

• CER (primary) — range: [0, 1]
  • Character Error Rate calculated as the Levenshtein edit distance between the predicted transcription and the ground truth, normalized by the length of the ground truth string.
• WER — range: [0, 1]
  • Word Error Rate calculated as the Levenshtein edit distance between the predicted word sequence and the ground truth word sequence, normalized by the length of the ground truth word sequence.
• HCPR — range: [0, 1]
  • Historical Character Preservation Rate measuring the percentage of period-specific characters (e.g., i/ї, ъ) correctly preserved in the model's output relative to the source text.
• AIR — range: [0, 1]
  • Archaic Insertion Rate measuring the frequency of obsolete, pre-Petrine characters incorrectly inserted by the model that do not appear in the source document.
• CaseER — range: [0, 1]
  • Case Error Rate quantifying the proportion of case assignment mistakes, with specific emphasis on visually distinctive historical characters.

Input / output format

Input: Image of a single text line, full page, or sliding window of lines from an 18th-century Russian historical document.

Output: Transcribed text string corresponding to the input image content.

Scoring recipe

def compute_cer(pred, gold):
    dist = levenshtein_distance(pred, gold)
    return dist / len(gold) if len(gold) > 0 else 0.0

def compute_hcpr(pred, gold, period_chars):
    gold_period = [c for c in gold if c in period_chars]
    if not gold_period: return 0.0
    correct = sum(1 for c in gold_period if c in pred)
    return correct / len(gold_period)

def compute_air(pred, gold, archaic_chars):
    inserted = [c for c in pred if c in archaic_chars and c not in gold]
    return len(inserted) / len(pred) if pred else 0.0

Common pitfalls

• Standard metrics like CER and WER fail to detect 'over-historicization', where models systematically insert archaic characters from incorrect historical periods.
• Full-page processing inputs risk hallucinations or detail loss on dense layouts, making line-by-line or sliding-window modes more reliable for accuracy assessment.
• Case-insensitive evaluation masks critical orthographic errors, as historical Russian texts rely heavily on case distinctions for period-specific spelling.

Evidence (verbatim from paper)

We employed an evaluation framework with multiple metrics to assess OCR accuracy, historical fidelity, and case sensitivity: Standard OCR Metrics. Character Error Rate (CER) and Word Error Rate (WER), using Levenshtein distance between prediction and ground truth. Case-Insensitive Metrics. CER and WER after lowercasing, to isolate character recognition from case errors. Historical Fidelity Metrics. Historical Character Preservation Rate (HCPR) for period-specific characters (i/ї, , ъ); Archaic Insertion Rate (AIR) for insertion of obsolete, pre-Petrine characters.

Citation

@misc{levchenko2025historical,
  title={Evaluating LLMs for Historical Document OCR: A Methodological Framework for Digital Humanities},
  author={Levchenko (2025)},
  year={2025},
  note={arXiv:2510.06743}
}

• arXiv: 2510.06743