Olmocr Bench Eval

olmocr-bench-eval

LightOnOCR: A 1B End-to-End Multilingual Vision-Language Model for State-of-the-Art OCR — Taghadouini et al. (2026) (arXiv:2601.14251, 2026)

What this evaluates

Evaluates end-to-end OCR models on full-page transcription accuracy across diverse document types (academic papers, old scans, math, tables, multi-column layouts) and tests their ability to localize embedded images via bounding box prediction.

Datasets

• OlmOCR-Bench — total 1403; splits: test (1403)
• LightOnOCR-bbox-bench — total 855; splits: test (855)

Metrics

• Overall Score (primary) — range: percent
  • Average character/token accuracy across document categories (ArXiv, Old Scans, Math, Tables, Multi-column, Long, Tiny Text). Headers/footers are explicitly excluded from scoring to align with full-page transcription objectives.
• F1@0.5 — range: [0, 1]
  • F1 score for bounding box detection computed at an Intersection over Union (IoU) threshold of 0.5.
• IoU — range: [0, 1]
  • Mean Intersection over Union between predicted and ground truth bounding boxes for visual elements.
• Count Accuracy — range: percent
  • Percentage of pages where the model correctly predicts the exact number of visual elements (figures/images).
• Throughput (pages/sec) — range: other
  • Total number of pages processed divided by the wall-clock inference time to complete the benchmark.

Input / output format

Input: High-resolution document images (single pages)

Output: Transcribed text (and bounding box coordinates for localization variants)

Scoring recipe

def compute_ocr_score(predictions, gold):
    cat_scores = []
    for cat in ['ArXiv', 'Old Scans', 'Math', 'Tables', 'Multi-column', 'Long', 'Tiny Text']:
        cat_scores.append(accuracy_score(predictions[cat], gold[cat]))
    return mean(cat_scores)  # Excludes headers/footers

def compute_bbox_metrics(pred_boxes, gt_boxes):
    f1 = f1_score_at_iou(pred_boxes, gt_boxes, iou_thresh=0.5)
    iou = mean_iou(pred_boxes, gt_boxes)
    count_acc = accuracy_score(len(pred_boxes), len(gt_boxes))
    return f1, iou, count_acc

Common pitfalls

• The original OlmOCR-Bench definition rewards omitting headers/footers, which conflicts with full-page transcription objectives; the authors explicitly exclude this category from scoring.
• Evaluation is strictly single-pass without test-time heuristics like rotation sweeps or retries, so results reflect raw model capability rather than post-processed performance.

Evidence (verbatim from paper)

We evaluate LightOnOCR-2-1B on OlmOCR-Bench as our primary OCR benchmark. We evaluate all models without test-time heuristics (e.g., retries or rotation correction) to reflect raw model behavior. Table 1 shows that LightOnOCR-2-1B achieves the highest overall score ($83.2\pm 0.9$) among evaluated systems, outperforming substantially larger end-to-end models while using only 1B parameters and being end-to-end trainable.

Citation

@misc{taghadouini2026lightonocr,
  title={LightOnOCR: A 1B End-to-End Multilingual Vision-Language Model for State-of-the-Art OCR},
  author={Taghadouini et al. (2026)},
  year={2026},
  note={arXiv:2601.14251}
}

• arXiv: 2601.14251