Videogameqa Bench Eval

videogameqa-bench-eval

VideoGameQA-Bench: Evaluating Vision-Language Models for Video Game Quality Assurance — Taesiri et al. (2025) (arXiv:2505.15952, 2025)

What this evaluates

Evaluates vision-language models on video game quality assurance tasks, including glitch detection, temporal reasoning, and bug reporting. It probes the model's ability to process sampled video frames, identify visual anomalies, and generate structured or descriptive reports about game glitches.

Datasets

• VideoGameQA-Bench — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: [0, 1]
  • Task-wise accuracy is computed for image and video tasks separately. The final score is the average of the image-task accuracy and video-task accuracy. Open-ended bug reports are scored via an LLM-as-a-judge (OpenAI o3) comparing outputs to ground truth references.

Input / output format

Input: Video clips sampled at 1 FPS (or lower for open-weight models) or single frames, accompanied by benchmark questions.

Output: Valid JSON format for standard benchmark questions. Descriptive text for bug-reporting tasks.

Scoring recipe

def compute_final_score(predictions, golds, task_types):
    img_correct, img_total = 0, 0
    vid_correct, vid_total = 0, 0
    for pred, gold, task in zip(predictions, golds, task_types):
        if task == 'bug_report':
            score = llm_judge.evaluate(pred, gold)
        elif is_valid_json(pred) and pred == gold:
            score = 1.0
        else:
            score = 0.0
        if task in ['image']: img_correct += score; img_total += 1
        else: vid_correct += score; vid_total += 1
    img_acc = img_correct / img_total if img_total else 0
    vid_acc = vid_correct / vid_total if vid_total else 0
    return (img_acc + vid_acc) / 2

Common pitfalls

• Models must output strictly valid JSON; even minor formatting errors cause the response to be disregarded entirely.
• Video inputs are downsampled to 1 FPS, which may obscure fast-moving glitches in some cases.
• Open-ended bug reports require an LLM-as-a-judge (OpenAI o3) rather than exact string matching, introducing potential judge bias.

Evidence (verbatim from paper)

Valid JSON output: All benchmark questions explicitly require models to output responses in a valid JSON format. Any responses not in JSON or containing malformed JSON structures will be disregarded, even if the model's output is only slightly different from the ground truth label. ... We ranked models by averaging accuracies across image and video tasks. Task-wise accuracies were first averaged within each type, then combined for the final score.

Citation

@misc{taesiri2025videogameqabench,
  title={VideoGameQA-Bench: Evaluating Vision-Language Models for Video Game Quality Assurance},
  author={Taesiri et al. (2025)},
  year={2025},
  note={arXiv:2505.15952}
}

• arXiv: 2505.15952