qhjqhj00 mcqa-accuracy-eval
HELM: Hyperbolic Large Language Models via Mixture-of-Curvature Experts — Neil He et al. (2025) (arXiv:2505.24722, 2025)
What this evaluates
Evaluates the ability of hyperbolic and Euclidean LLMs to answer multiple-choice questions across STEM, general knowledge, and commonsense reasoning domains. It probes how well the models capture semantic hierarchies and perform complex reasoning under few-shot and zero-shot settings.
Datasets
- MMLU — total ?; splits: test (-1)
- ARC-Challenging — total ?; splits: test (-1)
- CommonsenseQA — total ?; splits: test (-1)
- HellaSwag — total ?; splits: test (-1)
- OpenbookQA — total ?; splits: test (-1)
Metrics
accuracy(primary) — range: percent- Percentage of correctly answered multiple-choice questions. Calculated as (number of correct predictions / total number of questions) * 100.
Input / output format
Input: Multiple-choice question prompts formatted with 5-shot demonstrations for MMLU and ARC-Challenging, or 0-shot for CommonsenseQA, OpenbookQA, and HellaSwag. Tokens are processed using the LLaMA3.1-8B tokenizer (128K vocab).
Output: A single selected option letter corresponding to the correct answer from the provided choices.
Scoring recipe
def compute_accuracy(predictions, gold_labels):
correct = sum(1 for pred, gold in zip(predictions, gold_labels) if pred == gold)
return (correct / len(gold_labels)) * 100
Common pitfalls
- Shot settings differ by dataset: 5-shot for MMLU and ARC-Challenging, but 0-shot for CommonsenseQA, OpenbookQA, and HellaSwag.
- 100M models report performance averaged over 3 runs with standard deviation, while 1B models report single runs without standard deviation due to compute constraints.
- Baselines (LLaMA, DeepSeekV3) are trained from scratch with identical data and hyperparameters, not evaluated as frozen pretrained models.
Evidence (verbatim from paper)
We evaluate both HELM-MiCE and HELM-D at 100M-parameter scales, across a variety of benchmarks spanning STEM problem-solving, general knowledge, and commonsense reasoning. The dense models also serve as an ablation comparison with the MiCE models. We further scale the HELM-MiCE to 1B parameters as the smaller HELM-MiCE model outperformed HELM-D overall. We report the accuracy of the models’ abilities to answer multiple choice questions from the benchmarks.
Citation
@misc{he2025helm,
title={HELM: Hyperbolic Large Language Models via Mixture-of-Curvature Experts},
author={Neil He et al. (2025)},
year={2025},
note={arXiv:2505.24722}
}
- arXiv: 2505.24722