LLM Benchmark Eval

llm-benchmark-eval

Large Language Diffusion Models — Nie et al. (2025) (arXiv:2502.09992, 2025)

What this evaluates

Evaluates a language model's general capabilities, including in-context learning, instruction following, mathematical reasoning, code generation, and bidirectional reversal reasoning across a suite of standard and custom benchmarks.

Datasets

• MMLU — total ?; splits: test (-1)
• GSM8K — total ?; splits: test (-1)
• HumanEval — total ?; splits: test (-1)
• Chinese Poem Sentence Pairs — total 496; splits: test (496)

Metrics

• accuracy (primary) — range: percent
  • Percentage of correctly answered questions or completed poem lines. Evaluated with specified in-context learning shots (e.g., 5-shot for MMLU, 4-shot for GSM8K, 0-shot for ARC-C).

Input / output format

Input: Task-specific prompt containing N-shot in-context examples followed by the test instance, or a single poem line for reversal tasks.

Output: Model-generated answer, selected option, or poem line.

Scoring recipe

predictions = model.generate(prompt)
gold = ground_truth
score = 100 * (sum(1 for p, g in zip(predictions, golds) if exact_match(p, g)) / len(golds))

Common pitfalls

• Evaluation protocols follow existing studies, but exact prompt templates and few-shot selection vary per benchmark (e.g., 0-shot for ARC-C, 5-shot for MMLU).
• Data leakage is a concern; the authors explicitly checked GSM8K but note closed-source dataset distributions may affect comparability.
• Reversal tasks are evaluated zero-shot without fine-tuning, so performance reflects inherent bidirectional capability rather than task-specific adaptation.

Evidence (verbatim from paper)

To comprehensively evaluate the in-context learning and instruction-following capabilities of LLaDA 8B, we conducted detailed comparisons with existing LLMs of similar scale. Task selection and evaluation protocols followed existing studies, covering popular benchmarks in general tasks, mathematics, code, and Chinese. To quantify the reversal reasoning ability of models, we follow the protocol established in Allen-Zhu and Li [[35]]. Specifically, we construct a dataset of 496 famous Chinese poem sentence pairs. Given a sentence from a poem, models are tasked with generating the subsequent line (forward) or the preceding line (reversal) without additional fine-tuning.

Citation

@misc{nie2025largelanguagediffusionmodels,
  title={Large Language Diffusion Models},
  author={Nie et al. (2025)},
  year={2025},
  note={arXiv:2502.09992}
}

• arXiv: 2502.09992