Llama4 Benchmark Eval

llama4-benchmark-eval

The Llama 4 Herd: Architecture, Training, Evaluation, and Deployment Notes — Adcock et al. (2026) (arXiv:2601.11659, 2026)

What this evaluates

Evaluates the language understanding, reasoning, coding, multilingual, and multimodal capabilities of Llama 4 models across a standardized suite of academic and industry benchmarks. It measures performance on text-only, code, and vision-language tasks using few-shot or zero-shot prompting protocols.

Datasets

• MMLU — total ?; splits: test (-1)
• MMLU-Pro — total ?; splits: test (-1)
• MATH — total ?; splits: test (-1)
• MBPP — total ?; splits: test (-1)
• LiveCodeBench — total ?; splits: test (-1)
• GPQA Diamond — total ?; splits: test (-1)
• ChartQA — total ?; splits: test (-1)
• DocVQA — total ?; splits: test (-1)
• MMMU — total ?; splits: test (-1)
• MTOB — total ?; splits: test (-1)

Metrics

• macro_avg/acc (primary) — range: [0, 1]
  • Macro-averaged accuracy across all categories or tasks. Computed as the mean of per-category accuracy scores.
• em — range: [0, 1]
  • Exact-match accuracy. Returns 1 if the model's predicted answer exactly matches the ground truth, 0 otherwise.
• pass@1 — range: [0, 1]
  • Probability that the first generated solution passes all provided unit tests or satisfies the problem constraints.
• relaxed_accuracy — range: [0, 1]
  • Accuracy metric for chart understanding that allows for minor formatting or textual variations in the answer while preserving semantic correctness.
• ANLS — range: [0, 1]
  • Average Normalized Levenshtein Similarity. Computes the Levenshtein distance between predicted and ground truth text, normalized by the length of the ground truth, averaged across samples.
• chrF — range: [0, 100]
  • Character n-gram F-score. Measures the harmonic mean of character n-gram precision and recall between the generated translation and the reference, commonly used for long-context machine translation evaluation.

Input / output format

Input: Text prompts with optional images or code snippets, formatted with few-shot examples where specified (e.g., 5-shot for MMLU/MMLU-Pro, 4-shot for MATH, 3-shot for MBPP, 1-shot for TyDiQA, 0-shot for others). Long-context tasks use full-book or half-book documents up to 128K context window.

Output: Natural language answers, numerical values, or executable code blocks, depending on the benchmark. Multimodal tasks require text responses grounded in provided images.

Scoring recipe

def compute_metrics(predictions, golds, metric_name):
    if metric_name == 'macro_avg/acc':
        return np.mean([np.mean(p == g) for p, g in zip(predictions, golds)])
    elif metric_name == 'em':
        return np.mean([1.0 if p.strip() == g.strip() else 0.0 for p, g in zip(predictions, golds)])
    elif metric_name == 'pass@1':
        return np.mean([1.0 if passes_unit_test(p) else 0.0 for p in predictions])
    elif metric_name == 'relaxed_accuracy':
        return np.mean([1.0 if is_semantically_equivalent(p, g) else 0.0 for p, g in zip(predictions, golds)])
    elif metric_name == 'ANLS':
        return np.mean([levenshtein_sim(p, g) for p, g in zip(predictions, golds)])
    elif metric_name == 'chrF':
        return compute_chrf_score(predictions, golds)
    return 0.0

Common pitfalls

• Results are exclusively reported on bf16 precision; mixed precision or quantized evaluations are not included and may differ significantly.
• Multimodal baselines are excluded from comparison due to lack of multimodal support, making direct performance benchmarking against prior Llama versions impossible for vision tasks.
• Long-context evaluation uses MTOB with a fixed 128K context window, which may not reflect true infinite-context capabilities or retrieval accuracy beyond that limit.

Evidence (verbatim from paper)

Long-context evaluation is presented via MTOB with chrF scores in two translation directions, with the model card explicitly contrasting against a 128K context baseline

Citation

@misc{adcock2026llama4,
  title={The Llama 4 Herd: Architecture, Training, Evaluation, and Deployment Notes},
  author={Adcock et al. (2026)},
  year={2026},
  note={arXiv:2601.11659}
}

• arXiv: 2601.11659