Moe Routing Eval

moe-routing-eval

Improving Routing in Sparse Mixture of Experts with Graph of Tokens — Tam Nguyen et al. (2025) (arXiv:2505.00792, 2025)

What this evaluates

Evaluates the routing stability, robustness, and task performance of Sparse Mixture of Experts (SMoE) models enhanced with similarity or attention-aware mechanisms. It probes how token-level routing decisions affect language modeling, image classification, and downstream fine-tuning under both clean and perturbed conditions.

Datasets

• Wikitext-103 — total ?; splits: val (-1), test (-1)
• ImageNet-1K — total ?; splits: train (-1), val (-1), test (-1)
• ImageNet-C — total ?; splits: test (-1)
• ImageNet-A — total ?; splits: test (-1)
• ImageNet-R — total ?; splits: test (-1)
• ImageNet-O — total ?; splits: test (-1)
• SST5 — total ?; splits: test (-1)
• SST2 — total ?; splits: test (-1)
• Banking-77 — total ?; splits: test (-1)

Metrics

• perplexity (primary) — range: other
  • Exponential of the negative average log-likelihood of the ground-truth tokens: exp(-1/N * Σ log p(y_i | y_<i)). Lower values indicate better language modeling performance.
• Top-1 accuracy (primary) — range: [0, 1]
  • The proportion of correctly predicted class labels out of the total number of instances: sum(preds == gold) / N. Higher values indicate better classification or fine-tuning performance.
• routing fluctuation rate — range: percent
  • The percentage of tokens that change their assigned expert between consecutive training epochs (specifically epochs 59 and 60). Lower values indicate more stable routing.
• decision entropy rate — range: other
  • The average entropy of the probability distribution over experts for each token routing decision. Lower values indicate more confident and consistent expert selection.
• load balancing score — range: other
  • Measures the uniformity of token distribution across experts, typically evaluated via variance or standard deviation of expert assignment counts. More uniform distribution indicates better load balancing.

Input / output format

Input: Tokenized text sequences for language modeling and fine-tuning tasks; image patches processed through a Vision MoE (V-MoE) architecture for classification tasks.

Output: For LM: probability distribution over the vocabulary for the next token. For classification/fine-tuning: predicted class label.

Scoring recipe

# Perplexity
ppl = exp(-mean(log(probs[range(len(tokens)), tokens])))
# Accuracy
acc = sum(preds == gold) / len(gold)
# Fluctuation
fluct = count(tokens_changed_expert_between_epoch_59_and_60) / total_tokens
# Entropy
entropy = mean(-sum(p_expert_given_token * log(p_expert_given_token)))

Common pitfalls

• Confusing clean vs. attacked dataset results when comparing perplexity scores.
• Misinterpreting lower perplexity as worse performance (it is better).
• Overlooking that routing fluctuation and entropy are internal diagnostic metrics, not task performance metrics.
• Assuming the method replaces the base MoE architecture rather than acting as a plug-and-play enhancement to baselines like GLAM, X-MoE, and SMoE-dropout.

Evidence (verbatim from paper)

The models are evaluated on validation and test sets using perplexity scores (lower is better) on clean data and in adversarial scenarios, i.e. under word-swap attacks.

Citation

@misc{nguyen2025improvingrouting,
  title={Improving Routing in Sparse Mixture of Experts with Graph of Tokens},
  author={Tam Nguyen et al. (2025)},
  year={2025},
  note={arXiv:2505.00792}
}

• arXiv: 2505.00792