Marioqa Eval

marioqa-eval

MarioQA: Answering Questions by Watching Gameplay Videos — Mun et al. (2016) (arXiv:1612.01669, 2016)

What this evaluates

Evaluates a model's ability to perform video question answering with varying levels of temporal reasoning complexity. It probes whether models can correctly link visual events in gameplay videos to answer questions that require single-frame, event-level, or multi-step causal/temporal understanding.

Datasets

• MarioQA — total ?; splits: train (-1), test (-1)

Metrics

• accuracy (primary) — range: [0, 1]
  • Computed as the ratio of correct answers to the total number of questions. Predictions are compared against ground-truth answer classes from a fixed vocabulary of 57 classes.

Input / output format

Input: A gameplay video clip and a natural language question.

Output: A single answer class from a fixed vocabulary of 57 classes.

Scoring recipe

def compute_accuracy(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
    return correct / len(gold_labels)

Common pitfalls

• Confusing the training subset combinations (NT, NT+ET, NT+ET+HT) with the test subsets; models are always evaluated on the test split of each individual subset (NT, ET, HT).
• Assuming performance gains from adding ET/HT to training are solely due to increased data volume; the authors explicitly control for this by training with roughly equal example counts (Table 5).
• Using baseline models that lack one modality (Video Only or Question Only) as strong comparators, as they inherently miss critical information for VideoQA.

Evidence (verbatim from paper)

we evaluate algorithms in all three subsets by simply computing the ratio of correct answers to the total number of questions.

Citation

@misc{mun2016marioqa,
  title={MarioQA: Answering Questions by Watching Gameplay Videos},
  author={Mun et al. (2016)},
  year={2016},
  note={arXiv:1612.01669}
}

• arXiv: 1612.01669