Churro Ds Eval

churro-ds-eval

CHURRO: Making History Readable with an Open-Weight Large Vision-Language Model for High-Accuracy, Low-Cost Historical Text Recognition — Semnani et al. (2025) (arXiv:2509.19768, 2025)

What this evaluates

Evaluates the ability of vision-language models and OCR systems to accurately transcribe historical documents, including both printed and handwritten text across diverse languages and scripts. It probes robustness to long-term document degradation, variable layouts, and long-context inputs in zero-shot and fine-tuned settings.

Datasets

• Churro-DS — total ?; splits: train (-1), test (-1); repo https://github.com/stanford-oval/Churro

Metrics

• normalized Levenshtein similarity (primary) — range: [0, 1]
  • 1 minus the character-level Levenshtein distance divided by the length of the longer string (prediction or gold). Bounded in [0, 1].

Input / output format

Input: High-resolution image of a historical document (printed or handwritten).

Output: Text transcription generated by the model.

Scoring recipe

def normalized_levenshtein_similarity(pred: str, gold: str) -> float:
    dist = levenshtein_distance(pred, gold)
    max_len = max(len(pred), len(gold))
    if max_len == 0:
        return 1.0
    return 1.0 - (dist / max_len)

Common pitfalls

• Forgetting to normalize Arabic script diacritics and hamza using PyArabic before scoring.
• Averaging scores across language clusters versus computing a global average; the paper reports both but highlights the cluster average.
• Not applying identical normalization procedures to all model predictions, which can unfairly penalize certain systems.
• Ignoring context length limitations; models with insufficient context windows (e.g., DeepSeek-VL2) are excluded because they cannot process the long texts.

Evidence (verbatim from paper)

Specifically, we compute the character-level Levenshtein distance, which measures the number of insertions, deletions, and substitutions required to transform one string into another. We normalize this distance by the length of the longer string and convert it to similarity by subtracting it from 1, yielding a score in [0, 1]. This metric closely aligns with the Character Recognition Rate (1 minus Character Error Rate) commonly used in OCR studies, but is bounded within [0, 1], enabling comparison across varying text lengths. We refer to this metric as normalized Levenshtein similarity.

Citation

@misc{semnani2025churro,
  title={CHURRO: Making History Readable with an Open-Weight Large Vision-Language Model for High-Accuracy, Low-Cost Historical Text Recognition},
  author={Semnani et al. (2025)},
  year={2025},
  note={arXiv:2509.19768}
}

• arXiv: 2509.19768