qhjqhj00 muharaf-htr-eval
Muharaf: Manuscripts of Handwritten Arabic Dataset for Cursive Text Recognition — Saeed et al. (2024) (arXiv:2406.09630, 2024)
What this evaluates
Evaluates handwritten text recognition (HTR) systems on historical Arabic manuscripts. It probes the model's ability to accurately transcribe cursive text at both the page and line levels, handling contextual character variations and layout structures.
Datasets
- Muharaf — total 1646; splits: train (1500), val (50), test (96); repo https://github.com/mehreenmehreen/muharaf
Metrics
CER(primary) — range: [0, 1]- Character Error Rate computed as the Levenshtein distance between the predicted and ground truth strings, normalized by the length of the ground truth string.
WER— range: [0, 1]- Word Error Rate computed as the Levenshtein distance between the predicted and ground truth strings, normalized by the length of the ground truth string.
Input / output format
Input: Page images of handwritten Arabic manuscripts, and pre-segmented text line images for line-level evaluation.
Output: Recognized text string (sequence of characters or words) corresponding to the input image or line.
Scoring recipe
def compute_cer(pred, gt):
dist = levenshtein_distance(pred, gt)
return dist / len(gt)
def compute_wer(pred, gt):
dist = levenshtein_distance(pred, gt)
return dist / len(gt)
# Average over test set
cer = sum(compute_cer(p, g) for p, g in zip(predictions, ground_truths)) / len(ground_truths)
wer = sum(compute_wer(p, g) for p, g in zip(predictions, ground_truths)) / len(ground_truths)
Common pitfalls
- Line-level CER/WER can be artificially inflated by lines containing only isolated numbers or single words, where a single error causes the error rate to jump to 1.0.
- Results are averaged over three random train/val/test splits, so single-run evaluations may not be reproducible without fixing the random seed or using the provided splits.
Evidence (verbatim from paper)
The metrics for evaluation are the character error rate (CER) and word error rate (WER), both computed using Levenshtein distance normalized by the length of the string representing the ground truth.
Citation
@misc{saeed2024muharaf,
title={Muharaf: Manuscripts of Handwritten Arabic Dataset for Cursive Text Recognition},
author={Saeed et al. (2024)},
year={2024},
note={arXiv:2406.09630}
}
- arXiv: 2406.09630