Blackswan Eval

blackswan-eval

Black Swan: Abductive and Defeasible Video Reasoning in Unpredictable Events — Chinchure et al. (2024) (arXiv:2412.05725, 2024)

What this evaluates

Evaluates vision-language models on abductive and defeasible reasoning in videos depicting unpredictable events. It probes the ability to infer hidden causes from limited visual cues and revise hypotheses when new evidence emerges, testing reasoning beyond simple statistical recall.

Datasets

• BlackSwanSuite — total 1655; splits: (unstated)

Metrics

• accuracy (primary) — range: [0, 1]
  • Proportion of correct predictions out of total predictions for multiple-choice and yes/no variants.
• CLIP Score — range: [0, 1]
  • Maximum pairwise cosine similarity between CLIP embeddings of all model-generated responses and reference explanations per question.
• LLM-Match — range: [1, 5]
  • Average 1-5 similarity rating from Llama 3.1 8B comparing each model response to its reference explanation.
• Human Evaluation — range: other
  • Average scores across four parameters: Correctness (1-5), Depth and Thoughtfulness (1-5), Level of Detail (1-5), and Visual Entailment (0-1).

Input / output format

Input: Video file or uniformly sampled frames paired with a task-specific question (multiple-choice, yes/no, or open-ended).

Output: Selected option (MCQ), binary answer (Y/N), or generated textual explanation/response.

Scoring recipe

# Accuracy (MCQ/Y/N)
correct = sum(1 for pred, gold in zip(predictions, golds) if pred == gold)
accuracy = correct / len(predictions)

# CLIP Score
clip_scores = []
for q_preds, q_refs in zip(predictions, golds):
    max_sim = max(cosine_similarity(clip_embed(p), clip_embed(r)) for p in q_preds for r in q_refs)
    clip_scores.append(max_sim)
clip_score = mean(clip_scores)

# LLM-Match
llm_scores = [llama3_1_8b_rate_similarity(pred, ref) for pred, ref in zip(predictions, golds)]
llm_match = mean(llm_scores)

Common pitfalls

• CLIP Score depends heavily on the specific CLIP model/version used for embeddings; scores are not directly comparable across different CLIP variants.
• LLM-Match requires the exact prompt and LLM (Llama 3.1 8B) specified in the paper; using a different LLM or prompt alters the 1-5 rating scale distribution.
• Human evaluation parameters are subjective and require strict annotation guidelines to maintain inter-annotator reliability.

Evidence (verbatim from paper)

We report models' accuracy on the MCQ and Y/N variants. The quality of outputs generated for the generative variants of the tasks is evaluated using a combination of CLIP-based and LLM-based metrics and human evaluation. Given the open-ended nature of Forecaster and Detective, we generate 3 responses for each question from each model. For Reporter, we only generate a single explanation, since the entire video is revealed to the model. CLIP Score. We embed each model-generated response and each reference explanation in CLIP [27], and compute a pair-wise similarity score. We report the maximum pairwise similarity for each question, since we want to reward models for coming up with any plausible explanation.

Citation

@misc{chinchure2024blackswan,
  title={Black Swan: Abductive and Defeasible Video Reasoning in Unpredictable Events},
  author={Chinchure et al. (2024)},
  year={2024},
  note={arXiv:2412.05725}
}

• arXiv: 2412.05725