Ltdr Eval

ltdr-eval

Long-Tailed Distribution-Aware Router For Mixture-of-Experts in Large Vision-Language Model — Cai et al. (2025) (arXiv:2507.01351, 2025)

What this evaluates

Evaluates the vision-language understanding and domain generalization capabilities of Mixture-of-Experts (MoE) models. It tests how well a long-tailed distribution-aware router preserves routing variance for vision tokens while maintaining load balancing for language tokens, impacting both accuracy and inference efficiency.

Datasets

• GQA — total ?; splits: test (-1)
• ScienceQA-IMG — total ?; splits: test (-1)
• TextVQA — total ?; splits: test (-1)
• POPE — total ?; splits: test (-1)
• MME — total ?; splits: test (-1)
• MMBench — total ?; splits: test (-1)
• MM-Vet — total ?; splits: test (-1)
• PACS — total ?; splits: test (-1)
• VLCS — total ?; splits: test (-1)
• Office-Home — total ?; splits: test (-1)
• DomainNet — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: percent
  • Percentage of correctly predicted answers or class labels out of the total number of instances.
• MME score — range: other
  • Raw aggregate score from the MME benchmark, combining perception and cognition sub-tasks.
• Avg accuracy — range: percent
  • Arithmetic mean of accuracy scores across all benchmarks except MME.
• training time (s) — range: other
  • Total wall-clock time in seconds required to complete training.
• inference time (s) — range: other
  • Total wall-clock time in seconds required to process all benchmark instances.

Input / output format

Input: For vision-language tasks: an image paired with a text question/prompt. For vision-only tasks: a single image.

Output: For vision-language tasks: a generated text answer. For vision-only tasks: a predicted class label.

Scoring recipe

def compute_accuracy(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if normalize(p) == normalize(g))
    return (correct / len(gold_labels)) * 100

def compute_avg_accuracy(accuracies):
    valid_accs = [a for a in accuracies if a != 'MME_score']
    return sum(valid_accs) / len(valid_accs)

Common pitfalls

• MME reports a raw score (e.g., 1363.5) rather than a percentage, so it should not be averaged with other accuracy metrics.
• The 'Avg' column explicitly excludes the MME benchmark from the average calculation.
• Results marked with an asterisk (∗) indicate potential overlap with the model's training data, which may inflate reported accuracy.

Evidence (verbatim from paper)

As shown in Tab.[1], we evaluate the performance of our method on three image question-answering benchmarks and four benchmark toolkits, reporting both accuracy and the parameter of LLMs. ... We calculate the average performance “Avg” across all datasets except for MME.

Citation

@misc{cai2025ltldr,
  title={Long-Tailed Distribution-Aware Router For Mixture-of-Experts in Large Vision-Language Model},
  author={Cai et al. (2025)},
  year={2025},
  note={arXiv:2507.01351}
}

• arXiv: 2507.01351