Qianfan Ocr Eval

qianfan-ocr-eval

Qianfan-OCR: A Unified End-to-End Model for Document Intelligence — Daxiang Dong et al. (arXiv:2603.13398, 2026)

What this evaluates

This evaluation probes a unified vision-language model's ability to perform end-to-end document intelligence, including specialized OCR, general text recognition, document understanding, and key information extraction across diverse document types and multilingual scenarios.

Datasets

• Omni-Doc-Bench v1.5 — total ?; splits: (unstated)
• OLMOCRBench — total ?; splits: (unstated)
• OCRBench — total ?; splits: (unstated)
• DocVQA — total ?; splits: (unstated)
• ChartQA — total ?; splits: (unstated)
• Nanonets KIE — total ?; splits: (unstated)

Metrics

• F1 score — range: percent
  • Harmonic mean of precision and recall for extracted key-value fields, calculated as 2 * (precision * recall) / (precision + recall).
• normalized accuracy (0-100) (primary) — range: percent
  • Raw accuracy or F1 scores are linearly scaled to a 0–100 range for consistent cross-benchmark comparison.

Input / output format

Input: Document images, optionally accompanied by text prompts, questions, or layout-as-thought triggers (⟨think⟩ tokens).

Output: Markdown-formatted text, structured layout representations (bounding boxes, element types, reading order), or extracted key-value fields depending on the benchmark.

Scoring recipe

def score_kie(predictions, gold):
    pred_fields = extract_fields(predictions)
    gold_fields = extract_fields(gold)
    tp = len(set(pred_fields) & set(gold_fields))
    precision = tp / len(pred_fields) if pred_fields else 0
    recall = tp / len(gold_fields) if gold_fields else 0
    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
    return f1 * 100  # Normalize to 0-100 scale

Common pitfalls

• Comparing end-to-end VLMs directly against specialized pipeline OCR systems without accounting for architectural differences and parameter counts.
• Assuming KIE scores are raw F1 without applying the paper's specified 0–100 normalization scale.

Evidence (verbatim from paper)

Nanonets KIE (real-world invoice and receipt parsing, measured by F1 score). All scores are normalized to a 0–100 scale.

Citation

@misc{dong2026qianfanocr,
  title={Qianfan-OCR: A Unified End-to-End Model for Document Intelligence},
  author={Daxiang Dong et al.},
  year={2026},
  note={arXiv:2603.13398}
}

• arXiv: 2603.13398