Mobile Mmlu Eval

mobile-mmlu-eval

Mobile-MMLU: A Mobile Intelligence Language Understanding Benchmark — Bsharat et al. (2025) (arXiv:2503.20786, 2025)

What this evaluates

Evaluates language models' understanding of mobile-specific domains and tasks under on-device constraints. It probes the models' ability to answer multiple-choice questions across 80 real-world mobile domains, emphasizing practical usability, privacy, and personalization in daily mobile interactions.

Datasets

• Mobile-MMLU — total 16186; splits: test (16186); repo https://github.com/VILA-Lab/Mobile-MMLU

Metrics

• accuracy (primary) — range: percent
  • Percentage of correctly answered multiple-choice questions. Calculated as (number of correct predictions / total number of questions) * 100.

Input / output format

Input: A multiple-choice question with four options (A, B, C, D) covering a specific mobile domain.

Output: The model must output exactly one of the four option letters (A, B, C, or D) corresponding to the correct answer.

Scoring recipe

correct = 0
for pred, gold in zip(predictions, gold_labels):
    if pred.strip().upper() == gold.strip().upper():
        correct += 1
accuracy = (correct / len(gold_labels)) * 100

Common pitfalls

• Answer order bias: Small LLMs are highly sensitive to the position of the correct answer (A/B/C/D), causing performance variance exceeding 10% if options are not systematically balanced or randomized.
• Model size does not guarantee performance: Smaller models (e.g., 3B parameters) can outperform larger ones (e.g., 8B parameters) on mobile-specific tasks due to architectural or training differences.
• Zero-shot evaluation: The benchmark is evaluated in a zero-shot setting without in-context examples, which may yield different results compared to few-shot practices common in other benchmarks.

Evidence (verbatim from paper)

Given that Mobile-MMLU and Mobile-MMLU-Pro consist entirely of multiple-choice questions, we focus on accuracy as our primary evaluation metric. This approach allows for objective comparison across models while maintaining consistency with existing benchmarking practices in the field.

Citation

@misc{bsharat2025mobilemmlu,
  title={Mobile-MMLU: A Mobile Intelligence Language Understanding Benchmark},
  author={Bsharat et al. (2025)},
  year={2025},
  note={arXiv:2503.20786}
}

• arXiv: 2503.20786