Lime Mmt47 Eval

lime-mmt47-eval

LiME: Lightweight Mixture of Experts for Efficient Multimodal Multi-task Learning — Kowsher et al. (2026) (arXiv:2604.02338, 2026)

What this evaluates

Evaluates the ability of lightweight Mixture of Experts (MoE) parameter-efficient fine-tuning methods to generalize across diverse multimodal tasks. It probes how well shared PEFT modules with expert modulation vectors capture task-specific specialization without learned routing parameters.

Datasets

• MMT-47 — total 158000; splits: train (158000), test (-1)

Metrics

• accuracy (primary) — range: percent
  • Percentage of correctly predicted labels out of total test instances. Reported as mean ± standard deviation across 5 random seeds.
• throughput — range: other
  • Training speed measured in samples processed per second on a single H100 GPU.
• trainable_parameters — range: other
  • Total count of trainable parameters in the model.

Input / output format

Input: Multimodal instances (text, image, or video) paired with ground-truth labels from the MMT-47 benchmark mixture.

Output: Predicted class labels or regression values for each instance.

Scoring recipe

correct = sum(1 for pred, gold in zip(predictions, gold_labels) if pred == gold)
accuracy = (correct / len(gold_labels)) * 100
mean_acc = np.mean([accuracy(seed) for seed in 5_seeds])
std_acc = np.std([accuracy(seed) for seed in 5_seeds])

Common pitfalls

• Performance is highly sensitive to the number of experts (E); accuracy peaks at E=4-5 and degrades with more experts due to insufficient data per expert.
• The routing threshold θ significantly impacts results; too low (θ→0) activates noisy experts, while too high (θ→1) discards useful secondary experts, with optimal around 0.7.
• Load balancing coefficients must be carefully tuned; zero balancing causes expert collapse, while excessive balancing (>1.0) suppresses natural specialization and hurts accuracy.

Evidence (verbatim from paper)

Figure 3(c-d) reports GLUE accuracy as we vary the number of experts from 1 to 10. Both LiME and MoELoRA peak at 3-5 experts (stars), suggesting that a moderate number of experts often provides the best balance between added capacity and how well experts can be trained.

Citation

@misc{kowsher2026lime,
  title={LiME: Lightweight Mixture of Experts for Efficient Multimodal Multi-task Learning},
  author={Kowsher et al. (2026)},
  year={2026},
  note={arXiv:2604.02338}
}

• arXiv: 2604.02338