Mixeval X Eval

mixeval-x-eval

MixEval-X: Any-to-Any Evaluations from Real-World Data Mixtures — Ni et al. (2024) (arXiv:2410.13754, 2024)

What this evaluates

Evaluates multi-modal and agent models across any-to-any generation and action-planning tasks using real-world data mixtures. It probes capabilities in vision-language understanding, audio-language understanding, text-to-media generation, and API-level action planning.

Datasets

• MixEval-X — total ?; splits: general (-1), hard (-1)

Metrics

• accuracy (primary) — range: percent
  • Percentage of correctly generated responses for MMU tasks (Image2Text, Video2Text, Audio2Text).
• Elo score — range: other
  • Paired comparison scores derived from crowd-sourced human pairwise rankings using the Bradley-Terry model for MMG tasks (Text2Image, Text2Video, Text2Audio).

Input / output format

Input: Varies by task: MMU tasks receive image/video/audio plus text prompts; MMG tasks receive text prompts; Agent tasks receive text/image plus text prompts describing the environment and user goal.

Output: Varies by task: MMU tasks require text responses; MMG tasks require generated image/video/audio; Agent tasks require sequences of API-level actions.

Scoring recipe

if task in MMU_TASKS:
    score = 1.0 if prediction == gold else 0.0
    metric = mean(score)
elif task in MMG_TASKS:
    # Human pairwise ranking via MTurk
    # Bradley-Terry model converts pairwise wins/losses to Elo scores
    metric = compute_elo_score(pairwise_human_ratings)
elif task in AGENT_TASKS:
    # Task success based on API execution or action sequence validity
    metric = task_success_rate(predictions, environment_state)

Common pitfalls

• MMG tasks rely on human pairwise rankings rather than automatic metrics, which fail to capture output quality nuances.
• Input formatting must be standardized; MMG tasks require caption-like prompts rewritten by GPT-4, otherwise models may fail.
• For Video2Text, frame count limits significantly impact performance, especially on long videos.

Evidence (verbatim from paper)

Gemini 1.5 Pro ranking first at $62.7%$ accuracy on the general split but showing room for improvement on the Audio2Text-Hard split. ... The results are derived using the Bradley-Terry model, based on crowd-sourced user preferences.

Citation

@misc{ni2024mixevalx,
  title={MixEval-X: Any-to-Any Evaluations from Real-World Data Mixtures},
  author={Ni et al. (2024)},
  year={2024},
  note={arXiv:2410.13754}
}

• arXiv: 2410.13754