Olmocr Eval

olmocr-eval

olmOCR: Unlocking Trillions of Tokens in PDFs with Vision Language Models — Poznanski et al. (2025) (arXiv:2502.18443, 2025)

What this evaluates

Evaluates the ability of vision-language models and OCR tools to accurately linearize and extract structured content from complex, real-world PDFs. It probes reading order preservation, content comprehensiveness, and faithful representation of tables and equations.

Datasets

• olmOCR-mix-0225 — total ?; splits: eval (2017); repo https://github.com/allenai/olmocr

Metrics

• alignment (primary) — range: [0, 1]
  • Splits document into words, aligns them using Hirschberg’s algorithm, and calculates the proportion of matching words between the model output and the gold reference.
• ELO rating — range: ELO scale (base 1500)
  • Pairwise comparison score starting from a base of 1500. Computed as the average over 100 simulations to avoid ordering effects, with 95% confidence intervals via bootstrapping (5000 resamples).
• win rate — range: percent
  • Percentage of pairwise head-to-head comparisons where a model's output is preferred over another's based on human judgment.

Input / output format

Input: PDF page images (or rendered pages) paired with plain text outputs from competing linearization tools for side-by-side comparison; or raw PDF pages for automated alignment scoring.

Output: Linearized text representation of the PDF page (e.g., Markdown, LaTeX, or plain text) preserving reading order and structured elements.

Scoring recipe

def calc_alignment(gold, pred):
    g_words = tokenize(gold)
    p_words = tokenize(pred)
    aligned = hirschberg_align(g_words, p_words)
    return sum(1 for g, p in aligned if g == p) / len(g_words)

def calc_elo(judgments, base=1500, sims=100):
    ratings = {m: base for m in models}
    for _ in range(sims):
        for pair in judgments:
            ratings[pair.winner] += 32 * (1 - expected_score(ratings[pair.winner], ratings[pair.loser]))
    return np.mean([run_elo_sim(judgments) for _ in range(sims)])

def calc_win_rate(wins, losses):
    return wins / (wins + losses) * 100

Common pitfalls

• Alignment metric only measures word-level matching via Hirschberg's algorithm and ignores formatting differences (e.g., Markdown vs LaTeX).
• Human evaluation explicitly prioritizes 'usefulness for training LMs' over 'most faithful OCR', which may favor cleaner but slightly lossy outputs.
• ELO ratings are averaged over 100 simulations to mitigate ordering effects, so raw win rates alone do not reflect final rankings.

Evidence (verbatim from paper)

We calculate ELO ratings starting from a base of 1500 and report the average of 100 simulations to avoid ordering effects in ELO calculations; for 95% confidence intervals, we use bootstrapping with 5000 resamples.

Citation

@misc{poznanski2025olmocr,
  title={olmOCR: Unlocking Trillions of Tokens in PDFs with Vision Language Models},
  author={Poznanski et al. (2025)},
  year={2025},
  note={arXiv:2502.18443}
}

• arXiv: 2502.18443