Mbgvc Eval

mbgvc-eval

Knowledge Guided Entity-aware Video Captioning and A Basketball Benchmark — Zeyu Xi et al. (arXiv:2401.13888, 2024)

What this evaluates

Evaluates a model's ability to generate entity-aware, fine-grained text descriptions of basketball videos, specifically requiring accurate prediction of player names and precise action recognition.

Datasets

• MbgVC — total ?; splits: (unstated)

Metrics

• CIDEr — range: other
  • Computes cosine similarity of n-grams between generated and reference texts. Higher scores indicate better n-gram overlap and linguistic quality.
• METEOR — range: percent
  • Measures alignment between generated and reference text using synonymy, stemming, and exact match, with a penalty for fragmentation.
• Rouge-L — range: percent
  • Calculates the longest common subsequence between generated and reference texts to capture sentence-level structure.
• BLEU-4 — range: percent
  • Computes geometric mean of precision scores for 1- to 4-grams, with a brevity penalty to discourage overly short outputs.
• GDS (primary) — range: other
  • Game Description Score: a composite metric that jointly measures linguistic quality and name prediction accuracy. Higher scores indicate better overall performance, particularly in correctly generating player names and fine-grained actions.

Input / output format

Input: Video sequence (T=18 frames, 720×1280), cropped player images (120×180), and a list of candidate player names.

Output: Natural language caption describing the basketball action, including specific player names and fine-grained action types.

Scoring recipe

def compute_metrics(predictions, references):
    # Standard metrics use official implementations
    cider = cider_score(predictions, references)
    meteor = meteor_score(predictions, references)
    rouge_l = rouge_l_score(predictions, references)
    bleu4 = bleu4_score(predictions, references)
    
    # GDS is a composite of linguistic quality and name prediction accuracy
    # Exact formula not provided; computed as a weighted combination of n-gram overlap and entity match rate
    gds = composite_linguistic_and_name_score(predictions, references)
    
    return {'CIDEr': cider, 'METEOR': meteor, 'Rouge-L': rouge_l, 'BLEU-4': bleu4, 'GDS': gds}

Common pitfalls

• Concise dataset descriptions make standard n-gram metrics (BLEU, METEOR) less discriminative for performance differences.
• Models frequently confuse similar-looking actions (e.g., layups vs. close-range shots) or misjudge spatial distance (3-point vs. 2-point shots).
• Player name prediction is highly sensitive to training exposure; models without explicit name knowledge fail to generate accurate entities.

Evidence (verbatim from paper)

As shown in [Tab. 3], EAC outperforms other 3 models by a large margin on CIDEr. This notable performance can be attributed to the way CIDEr calculates cosine similarities of n-grams between the generated text and reference text. Highly accurate name prediction contributes to a higher cosine similarity between the n-grams of the generated text and the reference text. V2C, Clip4Caption, and SwinBert exhibit lower performance in accurately generating names, resulting in much lower CIDEr scores. Although EAC achieves the best performance across all metrics, the differences are not as pronounced in METEOR, Rouge-L, and BLEU. This can be attributed to the concise nature of the text descriptions in the dataset, primarily consisting of names and fine-grained actions. Notably, EAC outperforms SwinBert 31.3% on GDS and outperforms Clip4Caption 31.9% on GDS. The higher GDS indicates better overall model performance, particularly in name prediction.

Citation

@misc{xi2024knowledge,
  title={Knowledge Guided Entity-aware Video Captioning and A Basketball Benchmark},
  author={Zeyu Xi et al.},
  year={2024},
  note={arXiv:2401.13888}
}

• arXiv: 2401.13888