How do I install the Mmu RAG Arena Eval skill?

Run npx skillmds add qhjqhj00/mmu-rag-arena-eval in your terminal (requires Node.js), paste this page's agent-chat prompt into Claude, Cursor, or any MCP-connected agent, or download the SKILL.md file and copy it into your agent's skills directory.

Is Mmu RAG Arena Eval safe to use?

This skill has not completed SkillMD's automated safety review yet. Capability flags: makes network calls. SkillMD never runs a skill's scripts for you; review the SKILL.md before installing.

Which AI agents work with Mmu RAG Arena Eval?

This skill is tagged as working with Claude Code, Claude.ai, OpenAI Codex. SKILL.md is an open format, so most agents that read a skills directory can load it too.

Is Mmu RAG Arena Eval free to use?

Yes. Installing skills from SkillMD is free, and the skill stays under its author's original license.

Who published Mmu RAG Arena Eval?

qhjqhj00 (@qhjqhj00) published this skill. Their other Agent Skills are listed on their SkillMD profile.

Mmu RAG Arena Eval by qhjqhj00 — Agent Skill

Name: Mmu RAG Arena Eval
Rating: 4.5 (4 reviews)
Author: qhjqhj00

Mmu RAG Arena Eval

Evaluates RAG systems in a live, user-centric arena setting by routing queries to appropriate retrieval pipelines and measuring response quality through direct human feedback. It probes capabilities like retrieval grounding, synthesis coherence, response relevance, and appropriate verbosity in realistic deep-research scenarios. Use when the user wants to benchmark on MMU-RAG Competition / RAG Arena, or asks about evaluating this task. Reports Preference Ratio.

qhjqhj00 0e0cbce 2.6 KB Updated Sep 8, 2026 3 repo stars

File contents

mmu-rag-arena-eval

RMIT-ADM+S at the MMU-RAG NeurIPS 2025 Competition — Ran et al. (2026) (arXiv:2602.20735, 2026)

What this evaluates

Datasets

MMU-RAG Competition / RAG Arena — total ?; splits: test (-1); repo https://github.com/rmit-ir/NeurIPS-MMU-RAG

Metrics

Preference Ratio (primary) — range: [0, 1]
- Computed as the ratio of 👍 (thumbs up) to 👎 (thumbs down) responses submitted by participants during the evaluation session.

Input / output format

Input: User-submitted queries in a (semi-)realistic deep-research scenario, interacting with the system via a web interface.

Output: System-generated responses, followed by participant binary feedback (👍/👎), open-ended comments, and verbal reflections.

Scoring recipe

def calculate_preference_ratio(upvotes, downvotes):
    total = upvotes + downvotes
    if total == 0:
        return 1.0
    return upvotes / total

Common pitfalls

Relying solely on aggregate quantitative metrics may miss nuanced trade-offs in verbosity, grounding, and response structure that only emerge through live user interaction.
Synthetic LLM-generated queries and LLM-as-a-Judge evaluations can introduce distributional mismatch and evaluation biases, failing to capture realistic user failure modes.

Evidence (verbatim from paper)

We analyzed feedback for each query using two complementary approaches: (i) Preference Ratio, computed as the ratio of 👍 to 👎 responses, and (ii) manual inspection of open-ended comments to identify qualitative themes and contextual nuances.

Citation

@misc{ran2026rmitadm,
  title={RMIT-ADM+S at the MMU-RAG NeurIPS 2025 Competition},
  author={Ran et al. (2026)},
  year={2026},
  note={arXiv:2602.20735}
}

arXiv: 2602.20735

---name: mmu-rag-arena-evaldescription: Evaluates RAG systems in a live, user-centric arena setting by routing queries to appropriate retrieval pipelines and measuring response quality through direct human feedback. It probes capabilities like retrieval grounding, synthesis coherence, response relevance, and appropriate verbosity in realistic deep-research scenarios. Use when the user wants to benchmark on MMU-RAG Competition / RAG Arena, or asks about evaluating this task. Reports Preference Ratio.---# mmu-rag-arena-eval> RMIT-ADM+S at the MMU-RAG NeurIPS 2025 Competition — Ran et al. (2026) (arXiv:2602.20735, 2026)## What this evaluatesEvaluates RAG systems in a live, user-centric arena setting by routing queries to appropriate retrieval pipelines and measuring response quality through direct human feedback. It probes capabilities like retrieval grounding, synthesis coherence, response relevance, and appropriate verbosity in realistic deep-research scenarios.## Datasets- **MMU-RAG Competition / RAG Arena** — total ?; splits: test (-1); repo https://github.com/rmit-ir/NeurIPS-MMU-RAG## Metrics- `Preference Ratio` **(primary)** — range: [0, 1] - Computed as the ratio of 👍 (thumbs up) to 👎 (thumbs down) responses submitted by participants during the evaluation session.## Input / output format**Input**: User-submitted queries in a (semi-)realistic deep-research scenario, interacting with the system via a web interface.**Output**: System-generated responses, followed by participant binary feedback (👍/👎), open-ended comments, and verbal reflections.## Scoring recipe```pythondef calculate_preference_ratio(upvotes, downvotes): total = upvotes + downvotes if total == 0: return 1.0 return upvotes / total```## Common pitfalls- Relying solely on aggregate quantitative metrics may miss nuanced trade-offs in verbosity, grounding, and response structure that only emerge through live user interaction.- Synthetic LLM-generated queries and LLM-as-a-Judge evaluations can introduce distributional mismatch and evaluation biases, failing to capture realistic user failure modes.## Evidence (verbatim from paper)> We analyzed feedback for each query using two complementary approaches: (i) Preference Ratio, computed as the ratio of 👍 to 👎 responses, and (ii) manual inspection of open-ended comments to identify qualitative themes and contextual nuances.## Citation```bibtex@misc{ran2026rmitadm, title={RMIT-ADM+S at the MMU-RAG NeurIPS 2025 Competition}, author={Ran et al. (2026)}, year={2026}, note={arXiv:2602.20735}}```- arXiv: 2602.20735

Frequently asked questions