Openevents V1 Eval

openevents-v1-eval

OpenEvents V1: Large-Scale Benchmark Dataset for Multimodal Event Grounding — Nguyen et al. (2025) (arXiv:2506.18372, 2025)

What this evaluates

Probes a model's ability to perform multimodal event grounding by integrating visual content with textual context. It evaluates three interdependent capabilities: generating event-aware image captions, retrieving relevant news articles from images, and retrieving images from narrative event descriptions.

Datasets

• OpenEvents V1 — total ?; splits: test (-1)

Metrics

• CLIPScore (primary) — range: [0, 1]
  • Measures semantic alignment between image and text embeddings using a pre-trained CLIP model. Higher scores indicate better cross-modal consistency.
• CIDEr — range: [0, 1]
  • Consensus-based Image Description Evaluation; computes n-gram TF-IDF weighted overlap between generated and reference captions.
• BLEU-4 — range: [0, 1]
  • Bilingual Evaluation Understudy; calculates precision of 4-gram matches between prediction and references with brevity penalty.
• METEOR — range: [0, 1]
  • Evaluates translation quality using precision, recall, and penalties for fragmentation and synonymy.
• mAP (primary) — range: [0, 1]
  • Mean Average Precision; averages the precision at each recall threshold across all queries.
• NDCG — range: [0, 1]
  • Normalized Discounted Cumulative Gain; measures ranking quality by discounting gains logarithmically based on position.
• NN — range: [0, 1]
  • Nearest Neighbor accuracy; fraction of queries where the top-ranked retrieved item matches the ground truth.
• AUC — range: [0, 1]
  • Area Under the Precision-Recall Curve; summarizes the trade-off between precision and recall across thresholds.

Input / output format

Input: Task 1: Image (optionally paired with corresponding news article text). Task 2: Image. Task 3: Narrative caption describing a real-world event.

Output: Task 1: Text caption. Task 2: Ranked list of news articles. Task 3: Ranked list of images.

Scoring recipe

# Captioning (Task 1)
clip_score = cosine_sim(CLIP.encode(img), CLIP.encode(pred))
cider, bleu4, meteor = standard_metrics(refs, pred)

# Retrieval (Tasks 2 & 3)
scores = model_similarity(query, candidates)
ranked = argsort(scores, desc=True)
ap = average_precision(ranked, labels)
ndcg = normalized_dcg(ranked, labels)
nn_acc = 1.0 if ranked[0] == labels.argmax() else 0.0
auc = area_under_pr_curve(scores, labels)

Common pitfalls

• Models trained only on visual content (Image-only pipeline) significantly underperform because ground-truth captions contain event-specific details (named entities, timelines, causes) not visible in the image.
• Direct query-to-image retrieval struggles with complex, narrative-style captions that reflect deeper event semantics; hybrid approaches using intermediate article retrieval and re-ranking are necessary for competitive performance.
• Using OpenCLIP instead of CLIP yields lower retrieval accuracy on this dataset due to differences in training data curation (public datasets vs. carefully curated image-text correspondence).

Evidence (verbatim from paper)

To assess caption quality, we report standard captioning metrics including CLIPScore, CIDEr, BLEU-4 and METEOR, which measure fluency, descriptiveness, and semantic alignment with reference captions.

Citation

@misc{nguyen2025openevents,
  title={OpenEvents V1: Large-Scale Benchmark Dataset for Multimodal Event Grounding},
  author={Nguyen et al. (2025)},
  year={2025},
  note={arXiv:2506.18372}
}

• arXiv: 2506.18372