Ocr4mt Eval

ocr4mt-eval

OCR Improves Machine Translation for Low-Resource Languages — Ignat et al. (2022) (arXiv:2202.13274, 2022)

What this evaluates

Evaluates the performance of OCR systems on low-resource languages and scripts using both real and synthetically augmented PDF documents. It measures character-level accuracy to assess how OCR errors propagate and impact downstream tasks like machine translation.

Datasets

• OCR4MT — total 1659; splits: val (1659)

Metrics

• CER (primary) — range: [0, 1]
  • Character Error Rate (CER) is the minimum number of single-character edits (insertions, deletions, substitutions) required to transform the OCR output into the ground truth text, normalized by the length of the ground truth.

Input / output format

Input: PDF images of documents or individual articles containing text in various low-resource scripts.

Output: Transcribed text string corresponding to the content of the input PDF image.

Scoring recipe

def compute_cer(predictions, ground_truths):
    cer_scores = []
    for pred, gold in zip(predictions, ground_truths):
        edits = levenshtein_distance(pred, gold)
        cer_scores.append(edits / len(gold))
    return sum(cer_scores) / len(cer_scores)

Common pitfalls

• The benchmark mixes real PDFs (from UDHR) and artificially generated PDFs (from Flores 101), which have fundamentally different noise distributions and should be evaluated separately.
• Dataset construction removed ~7.8% of articles flagged as anomalies via a strict CER threshold (>2 standard deviations), which may bias the benchmark toward cleaner samples and underrepresent extreme OCR failure cases.

Evidence (verbatim from paper)

We automatically validate each article by measuring the CER per article. If the CER between the PDF labeled version and the text version is greater than two standard deviations away from the mean, the article is marked as anomalous (Cousineau and Chartier, 2010). We manually check and re-associate all the anomalous articles until no anomalies were detected. ... Finally, we obtain 1,659 pairs of PDF and corresponding text versions of articles.

Citation

@misc{ignat2022ocrimproves,
  title={OCR Improves Machine Translation for Low-Resource Languages},
  author={Ignat et al. (2022)},
  year={2022},
  note={arXiv:2202.13274}
}

• arXiv: 2202.13274