qhjqhj00 gamephysics-video-search-eval
CLIP meets GamePhysics: Towards bug identification in gameplay videos using zero-shot transfer learning — Taesiri et al. (2022) (arXiv:2203.11096, 2022)
What this evaluates
Evaluates zero-shot video retrieval capability using natural language queries to locate specific objects, compound descriptions, and game physics bugs in unstructured gameplay footage. It probes the model's ability to generalize across diverse open-world game genres and visual styles without fine-tuning.
Datasets
- GamePhysics — total 26954; splits: eval_subset (6192)
Metrics
top-k accuracy(primary) — range: percent- Binary measure: 100% if at least one relevant video appears in the top-k retrieved results, otherwise 0%. Averaged across all queries.
recall@5— range: percent- Proportion of relevant videos found among the top-5 retrieved results. Calculated as (number of relevant videos in top-5) / 5.
Input / output format
Input: Natural language text query paired with a collection of gameplay videos (processed via frame embeddings and indexed with Faiss).
Output: Ranked list of videos sorted by text-video similarity score.
Scoring recipe
def score_topk_accuracy(retrieved_videos, relevant_ids, k):
return 1.0 if any(v in relevant_ids for v in retrieved_videos[:k]) else 0.0
def score_recall_at_5(retrieved_videos, relevant_ids):
relevant_count = sum(1 for v in retrieved_videos[:5] if v in relevant_ids)
return relevant_count / 5.0
Common pitfalls
- Top-k accuracy is strictly binary (0% or 100%), so partial matches or multiple correct results within the top-k do not increase the score beyond 100%.
- Recall@5 uses a fixed denominator of 5 rather than the total number of ground-truth relevant videos, making it functionally equivalent to precision@5.
- Queries are manually curated per game and not standardized across the full dataset, which limits assessment of cross-domain generalization.
Evidence (verbatim from paper)
In the first and second experiments, we assess the sensitivity of our approach by measuring top-1 and top-5 accuracy. This is because for our approach to be useful to a game developer, the search system should be able to reliably identify objects specified in the text queries. Top-k accuracy is a binary measure; if there is a correct result in the top-k results, the accuracy is 100%, otherwise the accuracy is 0%... In the third experiment, we measured the accuracy of our approach using recall @5. Recall @5 is 100% when all five out of five retrieved videos match the bug query, 80% when four out of five retrieved videos match, etc. until 0% when there are no matching videos.
Citation
@misc{taesiri2022clipmeetsgamephysics,
title={CLIP meets GamePhysics: Towards bug identification in gameplay videos using zero-shot transfer learning},
author={Taesiri et al. (2022)},
year={2022},
note={arXiv:2203.11096}
}
- arXiv: 2203.11096