qhjqhj00 sard-ocr-eval
SARD: A Large-Scale Synthetic Arabic OCR Dataset for Book-Style Text Recognition — Nacar et al. (2025) (arXiv:2505.24600, 2025)
What this evaluates
Evaluates the robustness and accuracy of OCR models on synthetic, book-style Arabic documents with high typographic diversity across 10 fonts. It measures character-level precision, word-level accuracy, and overall sequence fluency to benchmark vision-language and traditional OCR systems.
Datasets
- SARD — total 843622; splits: test (2000)
Metrics
CER(primary) — range: [0, 1]- Character Error Rate: normalized Levenshtein edit distance between predicted and ground truth character sequences. Lower is better.
WER— range: [0, 1]- Word Error Rate: normalized Levenshtein edit distance between predicted and ground truth word sequences. Lower is better.
BLEU— range: [0, 1]- Bilingual Evaluation Understudy score: geometric mean of modified n-gram precisions with a brevity penalty to penalize overly short outputs. Higher is better.
Input / output format
Input: Full-page synthetic Arabic document images (book-style layout).
Output: Recognized text string corresponding to the page content.
Scoring recipe
def compute_cer(pred, ref):
return levenshtein_distance(pred, ref) / len(ref)
def compute_wer(pred, ref):
return levenshtein_distance(pred.split(), ref.split()) / len(ref.split())
def compute_bleu(pred, ref):
return nltk.translate.bleu_score.sentence_bleu([ref.split()], pred.split())
# Average scores across 2000 test images (200 per font)
Common pitfalls
- Performance varies significantly across different fonts, so averaging across fonts can mask font-specific weaknesses.
- BLEU score measures sequence fluency but may not perfectly align with character/word-level accuracy for OCR tasks.
- Synthetic dataset evaluation may not generalize to real-world scanned documents with noise, skew, or complex layouts.
Evidence (verbatim from paper)
To demonstrate the utility of SARD for benchmarking and to provide initial performance metrics, we evaluated several representative OCR models. Our evaluation centered on standard OCR metrics: Character Error Rate (CER), Word Error Rate (WER), and Bilingual Evaluation Understudy (BLEU) score. Lower CER and WER values signify better accuracy at the character and word levels, respectively. A higher BLEU score indicates greater overall sequence similarity between the predicted and ground truth text, a metric commonly used in text generation tasks but also valuable here for assessing the fluency and overall correctness of the recognized text block. ... For the results presented in Tables[2], [3], and [4], 200 randomly selected images were used for each of the ten fonts incorporated in the dataset, resulting in a total test set of 2000 images for this comprehensive analysis.
Citation
@misc{nacar2025sard,
title={SARD: A Large-Scale Synthetic Arabic OCR Dataset for Book-Style Text Recognition},
author={Nacar et al. (2025)},
year={2025},
note={arXiv:2505.24600}
}
- arXiv: 2505.24600