Shot2story Eval

shot2story-eval

Shot2Story: A New Benchmark for Comprehensive Understanding of Multi-shot Videos — Han et al. (2023) (arXiv:2312.10300, 2023)

What this evaluates

Evaluates multi-modal video understanding across three tasks: single-shot captioning, multi-shot summarization, and question answering. It probes a model's ability to process visual frames, optional ASR text, and shot-level structure to generate coherent descriptions or answer temporal, holistic, and audio-related questions.

Datasets

• Shot2Story — total 42958; splits: train (36951), val (1982), test (4025); repo https://github.com/bytedance/Shot2Story

Metrics

• BLEU@4 — range: [0, 1]
  • 4-gram precision with brevity penalty. Measures exact n-gram overlap between predicted and reference text.
• METEOR — range: [0, 1]
  • Harmonic mean of unigram precision and recall, weighted by penalty for fragmentation and synonym/stem matching.
• ROUGE — range: [0, 1]
  • Recall-oriented overlap of n-grams (typically ROUGE-L) between hypothesis and reference.
• CIDEr (primary) — range: other
  • Consensus-based Image Description Evaluation. Computes TF-IDF weighted n-gram similarity between predicted and reference captions to emphasize consensus with human references.
• Accuracy — range: percent
  • Percentage of correctly answered questions. Correctness is determined by a binary decision from GPT-3.5-turbo comparing the model's answer to the ground truth.

Input / output format

Input: Video frames sampled and resized to 224x224, optionally concatenated with ASR text transcripts. For QA tasks, the generated summary and a natural language question are provided as text prompts to an LLM.

Output: Text output: either a single-shot caption, a multi-shot video summary paragraph, or a direct answer to a question.

Scoring recipe

def score_captioning(pred, gold):
    bleu4 = compute_bleu(gold, pred, max_order=4)
    meteor = compute_meteor(gold, pred)
    rouge = compute_rouge_l(gold, pred)
    cider = compute_cider(gold, pred)
    return bleu4, meteor, rouge, cider

def score_qa(pred, gold, question):
    prompt = f'Q: {question}\nGold: {gold}\nPred: {pred}\nIs pred correct? (yes/no)'
    judge_response = llm_call('gpt-3.5-turbo', prompt)
    is_correct = 1 if 'yes' in judge_response.lower() else 0
    return is_correct

# Aggregate
accuracy = mean([score_qa(p, g, q) for p, g, q in qa_pairs])

Common pitfalls

• Confusing raw audio input with ASR text: the paper explicitly notes that ASR text is critical for performance, but adding raw audio features yields negligible gains.
• Overlooking shot-level structure: holistic models that ignore explicit shot boundaries underperform compared to shot-aware architectures, making shot processing essential for multi-shot tasks.
• QA evaluation relies on GPT-3.5-turbo for binary correctness rather than exact match, which may introduce judge bias or leniency compared to standard string-matching metrics.

Evidence (verbatim from paper)

For captioning and summarization, we evaluate the models using BLEU@4 (Papineni et al., 2002) (B), METEOR (Denkowski & Lavie, 2014) (M), ROUGE (Lin, 2004) (R), and CIDEr (Vedantam et al., 2015) (C).

Citation

@misc{han2023shot2story,
  title={Shot2Story: A New Benchmark for Comprehensive Understanding of Multi-shot Videos},
  author={Han et al. (2023)},
  year={2023},
  note={arXiv:2312.10300}
}

• arXiv: 2312.10300