Opencompass Downstream Eval

opencompass-downstream-eval

Harder Tasks Need More Experts: Dynamic Routing in MoE Models — Quzhe Huang et al. (arXiv:2403.07652, 2024)

What this evaluates

Evaluates language model performance across five diverse downstream benchmarks spanning commonsense reasoning, science QA, and complex reasoning. It specifically probes how dynamic expert routing mechanisms adapt to input difficulty compared to fixed Top-K routing.

Datasets

• PIQA — total ?; splits: test (-1)
• Hellaswag — total ?; splits: test (-1)
• ARC-e — total ?; splits: test (-1)
• CommonsenseQA — total ?; splits: test (-1)
• BBH — total ?; splits: test (-1)

Metrics

• accuracy (score) (primary) — range: percent
  • Percentage of correctly predicted answers or multiple-choice selections across all benchmark tasks, averaged across datasets.

Input / output format

Input: Multiple-choice or open-ended prompts for each downstream benchmark task.

Output: Model's predicted answer or selected choice.

Scoring recipe

correct = 0
total = 0
for pred, gold in zip(predictions, gold_labels):
    if pred == gold:
        correct += 1
    total += 1
accuracy = (correct / total) * 100

Common pitfalls

• Scores are averaged across heterogeneous benchmarks (PIQA, Hellaswag, ARC-e, CommonsenseQA, BBH), which can mask task-specific degradation.
• The dynamic routing threshold p is tuned at inference time on a pre-trained model rather than during fine-tuning, which may not reflect optimal training-time routing behavior.
• Different evaluation harnesses (e.g., OpenCompass vs. lm-evaluation-harness) may yield slightly different accuracy scores due to prompt formatting or answer extraction differences.

Evidence (verbatim from paper)

We use opencompass to evaluate our model. Table 1 shows the performance of different models on downstream tasks. Overall, the MoE models outperform the Dense models. Among all the MoE variants, our proposed Dynamic Adaptive MoE demonstrates the best performance, achieving at least a 0.7% higher score on average compared to other models.

Citation

@misc{huang2024harder,
  title={Harder Tasks Need More Experts: Dynamic Routing in MoE Models},
  author={Quzhe Huang et al.},
  year={2024},
  note={arXiv:2403.07652}
}

• arXiv: 2403.07652