Mop LLM Pruning Eval

mop-llm-pruning-eval

Compressing LLMs with MoP: Mixture of Pruners — Yamamoto et al. (2026) (arXiv:2602.06127, 2026)

What this evaluates

Evaluates the performance of pruned large language models on a suite of commonsense reasoning and multimodal benchmarks to measure accuracy retention under varying compression ratios.

Datasets

• ARC-e, ARC-c, HellaSwag, PIQA, WinoGrande, ScienceQA, VizWiz, LLaVA-Bench, MM-Vet — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: [0, 1]
  • Standard accuracy for WinoGrande; normalized accuracy for ARC-e, ARC-c, HellaSwag, and PIQA. Computed as the fraction of correctly answered multiple-choice questions.
• latency — range: seconds
  • Wall-clock time in seconds to process 12 input tokens and autoregressively generate 128 output tokens at batch size 1. Reported as speedup factor or percentage reduction relative to the dense baseline.

Input / output format

Input: Multiple-choice questions or prompts from the specified benchmarks, formatted according to EleutherAI LM Harness or LMMs-Eval conventions.

Output: Model's predicted answer choice or generated text response.

Scoring recipe

def compute_accuracy(predictions, golds):
    correct = sum(1 for p, g in zip(predictions, golds) if p == g)
    return correct / len(golds)
# Note: ARC-e/c, HellaSwag, PIQA use normalized accuracy; WinoGrande uses standard accuracy.

Common pitfalls

• Failing to distinguish between standard accuracy (WinoGrande) and normalized accuracy (ARC, HellaSwag, PIQA), which use different scoring conventions.
• Reporting inference latency without specifying the exact prompt length, output length, and batch size, as speedup is highly sensitive to these parameters.
• Using the calibration set (WikiText-2) for final performance reporting instead of the held-out evaluation benchmarks.

Evidence (verbatim from paper)

we evaluate the LLMs on five commonsense benchmarks: ARC-e / ARC-c, HellaSwag, PIQA, and WinoGrande. We employ the EleutherAI LM Harness framework to conduct these evaluations. Consistent with established literature, we report standard accuracy for WinoGrande and normalized accuracy for the remaining tasks.

Citation

@misc{yamamoto2026mop,
  title={Compressing LLMs with MoP: Mixture of Pruners},
  author={Yamamoto et al. (2026)},
  year={2026},
  note={arXiv:2602.06127}
}

• arXiv: 2602.06127