Ernie 5.0 Eval

ernie-5.0-eval

ERNIE 5.0 Technical Report — Haifeng Wang et al. (2026) (arXiv:2602.04705, 2026)

What this evaluates

Evaluates a trillion-parameter multimodal foundation model across text, vision, audio, and generation tasks to measure factual knowledge, reasoning, coding, instruction following, and agent capabilities.

Datasets

• PreciseWikiQA — total ?; splits: test (-1)
• MMLU-Pro — total ?; splits: test (-1)
• MATH — total ?; splits: test (-1)
• LiveCodeBench — total ?; splits: test (-1)
• MMMU-Pro — total ?; splits: test (-1)
• MathVista — total ?; splits: test (-1)
• GenEval — total ?; splits: test (-1)
• VBench — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: percent
  • Percentage of correctly predicted answers or task-specific scores normalized to [0, 100]. Different benchmarks use underlying metrics such as exact match, F1, or LLM-as-judge, but all results are reported as percentages.

Input / output format

Input: Task-specific prompts including text questions, images, audio, or video, often with few-shot examples (0 to 10 shots) and explicit instructions.

Output: Model-generated text responses, code, or generated images/videos depending on the benchmark task.

Scoring recipe

def compute_accuracy(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if normalize(p) == normalize(g))
    return (correct / len(gold)) * 100

Common pitfalls

• Shot counts vary widely across benchmarks (0-shot to 10-shot), and few-shot prompting strategies are not standardized.
• Different benchmarks use different underlying evaluation metrics (exact match, F1, automated generation metrics), but results are uniformly reported as percentages, obscuring task-specific nuances.
• Some generation benchmarks (e.g., GenEval, VBench) rely on automated metrics that may not fully correlate with human preference or real-world quality.

Evidence (verbatim from paper)

Table 1 summarizes the pre-training results of ERNIE 5.0 in comparison with strong open-source baselines on a diverse set of text benchmarks. Across these benchmarks, ERNIE 5.0 exhibits consistently strong and well-balanced performance in knowledge, reasoning, mathematics, coding, and multilingual tasks:

Citation

@misc{wang2026ernie5.0,
  title={ERNIE 5.0 Technical Report},
  author={Haifeng Wang et al. (2026)},
  year={2026},
  note={arXiv:2602.04705}
}

• arXiv: 2602.04705