# Marioqa Eval

> Evaluates a model's ability to perform video question answering with varying levels of temporal reasoning complexity. It probes whether models can correctly link visual events in gameplay videos to answer questions that require single-frame, event-level, or multi-step causal/temporal understanding. Use when the user wants to benchmark on MarioQA, or asks about evaluating this task. Reports accuracy.

- Skill: `qhjqhj00/marioqa-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/marioqa-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/marioqa-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/marioqa-eval

---


# marioqa-eval

> MarioQA: Answering Questions by Watching Gameplay Videos — Mun et al. (2016) (arXiv:1612.01669, 2016)

## What this evaluates

Evaluates a model's ability to perform video question answering with varying levels of temporal reasoning complexity. It probes whether models can correctly link visual events in gameplay videos to answer questions that require single-frame, event-level, or multi-step causal/temporal understanding.

## Datasets

- **MarioQA** — total ?; splits: train (-1), test (-1)

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Computed as the ratio of correct answers to the total number of questions. Predictions are compared against ground-truth answer classes from a fixed vocabulary of 57 classes.

## Input / output format

**Input**: A gameplay video clip and a natural language question.

**Output**: A single answer class from a fixed vocabulary of 57 classes.

## Scoring recipe

```python
def compute_accuracy(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
    return correct / len(gold_labels)
```

## Common pitfalls

- Confusing the training subset combinations (NT, NT+ET, NT+ET+HT) with the test subsets; models are always evaluated on the test split of each individual subset (NT, ET, HT).
- Assuming performance gains from adding ET/HT to training are solely due to increased data volume; the authors explicitly control for this by training with roughly equal example counts (Table 5).
- Using baseline models that lack one modality (Video Only or Question Only) as strong comparators, as they inherently miss critical information for VideoQA.

## Evidence (verbatim from paper)

> we evaluate algorithms in all three subsets by simply computing the ratio of correct answers to the total number of questions.

## Citation

```bibtex
@misc{mun2016marioqa,
  title={MarioQA: Answering Questions by Watching Gameplay Videos},
  author={Mun et al. (2016)},
  year={2016},
  note={arXiv:1612.01669}
}
```

- arXiv: 1612.01669