Mobileaibench Eval

mobileaibench-eval

MobileAIBench: Benchmarking LLMs and LMMs for On-Device Use Cases — Murthy et al. (2024) (arXiv:2406.10290, 2024)

What this evaluates

Evaluates the task performance, latency, and hardware utilization of quantized LLMs and LMMs on real mobile devices across standard NLP, multi-modal, and trust & safety benchmarks.

Datasets

• Databricks — total ?; splits: test (-1)
• HotpotQA — total ?; splits: test (-1)
• sql-create-context — total ?; splits: test (-1)
• CNN — total ?; splits: test (-1)
• XSum — total ?; splits: test (-1)
• VQA-v2 — total ?; splits: test (-1)
• GQA — total ?; splits: test (-1)
• VisWiz — total ?; splits: test (-1)
• TextVQA — total ?; splits: test (-1)
• SQA — total ?; splits: test (-1)
• AlpacaEval — total ?; splits: test (-1)
• MT-Bench — total ?; splits: test (-1)
• MMLU — total ?; splits: test (-1)
• GSM8K — total ?; splits: test (-1)
• TruthQA — total ?; splits: test (-1)
• BBQ — total ?; splits: test (-1)
• SC-101 — total ?; splits: test (-1)
• Adv-Inst — total ?; splits: test (-1)
• DNA — total ?; splits: test (-1)
• Priv-Lk — total ?; splits: test (-1)

Metrics

• Accuracy (primary) — range: [0, 1]
  • Proportion of correctly answered instances across VQA, MMLU, GSM8K, and Trust & Safety tasks.
• Exact Match (EM) — range: [0, 1]
  • Percentage of predictions that exactly match the ground truth answer.
• F1 Score (F1) — range: [0, 1]
  • Harmonic mean of precision and recall based on token overlap between prediction and gold.
• Rouge-1 (R1) — range: [0, 1]
  • Unigram overlap F-score between generated summary and reference.
• Rouge-L (RL) — range: [0, 1]
  • Longest common subsequence F-score between generated summary and reference.
• Win-Rate — range: [0, 1]
  • Proportion of times the model's response is preferred over a baseline in pairwise comparisons (AlpacaEval).
• MT-Bench Score — range: [0, 10]
  • Average score assigned by an LLM-as-a-judge across multi-turn conversation prompts.

Input / output format

Input: Text prompts for NLP tasks; image and text prompts for VQA tasks; instruction or multi-turn prompts for trust & safety and benchmark tasks.

Output: Model-generated text responses.

Scoring recipe

def compute_metrics(predictions, golds, task_type):
    if task_type in ['qa', 'vqa', 'trust_safety', 'mmlu', 'gsm8k']:
        return sum(1 for p, g in zip(predictions, golds) if p.strip() == g.strip()) / len(golds)
    elif task_type == 'summarization':
        return rouge1(predictions, golds), rougel(predictions, golds)
    elif task_type == 'text2sql':
        return sql_parser_score(predictions, golds), levenshtein_score(predictions, golds)
    elif task_type == 'alpaca':
        return pairwise_win_rate(predictions, golds)
    elif task_type == 'mtbench':
        return llm_judge_score(predictions, golds)

Common pitfalls

• Quantization at 3-bit causes severe performance drops for most LMMs, while some models (e.g., Moondream2) remain robust.
• Desktop evaluations may not capture real-world mobile constraints like thermal throttling, RAM limits, or actual latency (time-to-first-token).
• Different models and tasks exhibit varying sensitivity to quantization, so reporting only average scores can hide critical degradation patterns.

Evidence (verbatim from paper)

For question answering tasks, we employ Exact Match (EM) and F1 Score (F1). In the context of Text-to-SQL tasks, we utilize the SQL Parser (SP) and Levenshtein Score (LS). For summarization tasks, we measure performance using Rouge-1 (R1) and Rouge-L (RL). Additionally, we use Win-Rate for AlpacaEval, Score for MT Bench, and Accuracy for both MMLU and GSM8K.

Citation

@misc{murthy2024mobileaibench,
  title={MobileAIBench: Benchmarking LLMs and LMMs for On-Device Use Cases},
  author={Murthy et al. (2024)},
  year={2024},
  note={arXiv:2406.10290}
}

• arXiv: 2406.10290