# Mexa Eval

> Evaluates a training-free, dynamic multi-expert aggregation framework for multimodal reasoning. It tests the system's ability to select specialized pre-trained experts and synthesize their outputs across video, audio, 3D, and medical domains without fine-tuning. Use when the user wants to benchmark on Video-MMMU, MMAU, SQA3D, M3D, or asks about evaluating this task. Reports accuracy.

- Skill: `qhjqhj00/mexa-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/mexa-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/mexa-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Productivity
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/mexa-eval

---


# mexa-eval

> MEXA: Towards General Multimodal Reasoning with Dynamic Multi-Expert Aggregation — Yu et al. (2025) (arXiv:2506.17113, 2025)

## What this evaluates

Evaluates a training-free, dynamic multi-expert aggregation framework for multimodal reasoning. It tests the system's ability to select specialized pre-trained experts and synthesize their outputs across video, audio, 3D, and medical domains without fine-tuning.

## Datasets

- **Video-MMMU** — total ?; splits: test (-1)
- **MMAU** — total ?; splits: test (-1)
- **SQA3D** — total ?; splits: test (-1)
- **M3D** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly answered multiple-choice questions out of the total number of questions in the benchmark.

## Input / output format

**Input**: Multimodal inputs (video, audio, 3D scenes, or medical scans) paired with multiple-choice questions.

**Output**: A single selected option from the provided multiple-choice answers.

## Scoring recipe

```python
def calculate_accuracy(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
    return (correct / len(gold_labels)) * 100
```

## Common pitfalls

- The framework relies on external captioners for each modality; mismatched caption quality or prompt engineering can bottleneck performance regardless of the router/aggregator strength.
- Evaluation is strictly multiple-choice; open-ended generation or free-form reasoning capabilities are not measured.
- Performance is highly sensitive to the choice of router and aggregator (e.g., GPT-4o vs Qwen2.5-VL, DeepSeek vs GPT-4o); swapping these without re-evaluation may yield significantly different results.

## Evidence (verbatim from paper)

> We evaluate MEXA on all datasets under the multiple-choice QA setting, and report performance based on standard accuracy metrics across all experiments.

## Citation

```bibtex
@misc{yu2025mexa,
  title={MEXA: Towards General Multimodal Reasoning with Dynamic Multi-Expert Aggregation},
  author={Yu et al. (2025)},
  year={2025},
  note={arXiv:2506.17113}
}
```

- arXiv: 2506.17113