Moec Eval

moec-eval

MoEC: Mixture of Expert Clusters — Xie et al. (2022) (arXiv:2207.09094, 2022)

What this evaluates

Evaluates a Mixture of Experts model on cross-lingual machine translation and diverse natural language understanding tasks to measure how expert clustering and routing affect translation quality and classification accuracy.

Datasets

• WMT 2014 EN-DE + WMT-17 news-commentary-v12 — total 3960000; splits: train (3960000)
• GLUE (MNLI, CoLA, SST-2, QQP, QNLI, MRPC, STS-B) — total ?; splits: train (-1), val (-1), test (-1)

Metrics

• BLEU (primary) — range: percent
  • Standard n-gram overlap metric for machine translation, typically computed with sentence-level or corpus-level smoothing and tokenization.
• Accuracy — range: [0, 1]
  • Standard classification metric for GLUE tasks, calculated as the proportion of correctly predicted labels over total instances.

Input / output format

Input: Machine translation: source English sentence. NLU: Task-specific inputs (e.g., sentence pairs, single sentences, or prompts) depending on the GLUE subtask.

Output: Machine translation: predicted German sentence. NLU: predicted class label or regression score.

Scoring recipe

# For MT (WMT EN-DE)
bleu_score = compute_corpus_bleu(predictions, references)

# For GLUE tasks
acc = sum(1 for p, g in zip(preds, golds) if p == g) / len(golds)

# Report per-task accuracy; average across MNLI, CoLA, SST-2, QQP, QNLI, MRPC, STS-B (RTE excluded)

Common pitfalls

• Pre-training on BooksCorpus and English Wikipedia for 120k steps is required before GLUE fine-tuning, which significantly impacts reproducibility if not matched.
• The WMT training set combines WMT-14 with WMT-17 news-commentary-v12, differing from standard WMT-14-only baselines.
• RTE is explicitly excluded from the GLUE evaluation, so the standard 9-task average should not be used.

Evidence (verbatim from paper)

We name our model MoEC (Mixture of Expert Clusters), and evaluate the performance on bilingual machine translation and natural language understanding tasks. We use the X-MoE model from Chi et al. (2022) as our backbone architecture... The total training data contains 3.96M English-to-German sentence pairs. GLUE General Language Understanding Evaluation... is a collection of tools for evaluating the performance of models across a diverse set of existing NLU tasks, including MNLI, CoLA, SST-2, QQP, QNLI, MRPC and STS-B. It is worth mentioning that we will pre-train our model on the BooksCorpus and English Wikipedia corpus for 120k steps before fine-tuning on GLUE tasks.

Citation

@misc{xie2022moec,
  title={MoEC: Mixture of Expert Clusters},
  author={Xie et al. (2022)},
  year={2022},
  note={arXiv:2207.09094}
}

• arXiv: 2207.09094