Videogamebunny Eval

videogamebunny-eval

VideoGameBunny: Towards vision assistants for video games — Taesiri et al. (2024) (arXiv:2407.15295, 2024)

What this evaluates

Probes vision-language models' ability to understand video game contexts from screenshots, including recognizing actions, characters, UI elements, spatial relationships, and game mechanics. It evaluates how instruction-tuning on game-specific data improves performance compared to larger general-purpose models.

Datasets

• VideoGameBunny Dataset — total 185259; splits: train (-1), test (-1)

Metrics

• performance (primary) — range: other
  • Unspecified in the text; reported as mean performance and standard deviation across three independent runs, or as relative performance improvement (pp) compared to a baseline model.

Input / output format

Input: A screenshot from a video game paired with a natural language instruction or question.

Output: A natural language text response answering the question or following the instruction.

Scoring recipe

# Paper reports "performance" without specifying the exact metric.
# Standard VQA evaluation typically uses exact-match accuracy.
def score(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if p.strip().lower() == g.strip().lower())
    return correct / len(gold)
# Report mean and std across 3 runs; compute relative improvement (pp) vs baseline.

Common pitfalls

• The exact evaluation metric (e.g., exact-match accuracy, BLEU, or LLM-as-judge score) is not explicitly defined, only referred to as "performance".
• The paper does not clearly delineate the training/fine-tuning splits from the evaluation/test splits for the 185K image dataset.
• Experiments are conducted on a single NVIDIA A100 (80GB) GPU, which may constrain batch sizes and differ from standard multi-GPU reproducibility setups.

Evidence (verbatim from paper)

We evaluate different data mixing strategies at various sizes to see how both mixture and subset size change the performance of the model. We repeat each experiment three times, using different samples for each strategy to report the mean performance and standard deviation.

Citation

@misc{taesiri2024videogamebunny,
  title={VideoGameBunny: Towards vision assistants for video games},
  author={Taesiri et al. (2024)},
  year={2024},
  note={arXiv:2407.15295}
}

• arXiv: 2407.15295