# Openevents V1 Eval

> openevents-v1-eval

- Skill: `qhjqhj00/openevents-v1-eval` (Agent Skill)
- Install (CLI): `npx skillmds@latest add qhjqhj00/openevents-v1-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/openevents-v1-eval/raw
- Safety review: pending (external: skill-scanner PASS, skillspector PASS)
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-21
- Page: https://skillmd.com/skills/qhjqhj00/openevents-v1-eval

---


# openevents-v1-eval

> OpenEvents V1: Large-Scale Benchmark Dataset for Multimodal Event Grounding — Nguyen et al. (2025) (arXiv:2506.18372, 2025)

## What this evaluates

Probes a model's ability to perform multimodal event grounding by integrating visual content with textual context. It evaluates three interdependent capabilities: generating event-aware image captions, retrieving relevant news articles from images, and retrieving images from narrative event descriptions.

## Datasets

- **OpenEvents V1** — total ?; splits: test (-1)

## Metrics

- `CLIPScore` **(primary)** — range: [0, 1]
  - Measures semantic alignment between image and text embeddings using a pre-trained CLIP model. Higher scores indicate better cross-modal consistency.
- `CIDEr` — range: [0, 1]
  - Consensus-based Image Description Evaluation; computes n-gram TF-IDF weighted overlap between generated and reference captions.
- `BLEU-4` — range: [0, 1]
  - Bilingual Evaluation Understudy; calculates precision of 4-gram matches between prediction and references with brevity penalty.
- `METEOR` — range: [0, 1]
  - Evaluates translation quality using precision, recall, and penalties for fragmentation and synonymy.
- `mAP` **(primary)** — range: [0, 1]
  - Mean Average Precision; averages the precision at each recall threshold across all queries.
- `NDCG` — range: [0, 1]
  - Normalized Discounted Cumulative Gain; measures ranking quality by discounting gains logarithmically based on position.
- `NN` — range: [0, 1]
  - Nearest Neighbor accuracy; fraction of queries where the top-ranked retrieved item matches the ground truth.
- `AUC` — range: [0, 1]
  - Area Under the Precision-Recall Curve; summarizes the trade-off between precision and recall across thresholds.

## Input / output format

**Input**: Task 1: Image (optionally paired with corresponding news article text). Task 2: Image. Task 3: Narrative caption describing a real-world event.

**Output**: Task 1: Text caption. Task 2: Ranked list of news articles. Task 3: Ranked list of images.

## Scoring recipe

```python
# Captioning (Task 1)
clip_score = cosine_sim(CLIP.encode(img), CLIP.encode(pred))
cider, bleu4, meteor = standard_metrics(refs, pred)

# Retrieval (Tasks 2 & 3)
scores = model_similarity(query, candidates)
ranked = argsort(scores, desc=True)
ap = average_precision(ranked, labels)
ndcg = normalized_dcg(ranked, labels)
nn_acc = 1.0 if ranked[0] == labels.argmax() else 0.0
auc = area_under_pr_curve(scores, labels)
```

## Common pitfalls

- Models trained only on visual content (Image-only pipeline) significantly underperform because ground-truth captions contain event-specific details (named entities, timelines, causes) not visible in the image.
- Direct query-to-image retrieval struggles with complex, narrative-style captions that reflect deeper event semantics; hybrid approaches using intermediate article retrieval and re-ranking are necessary for competitive performance.
- Using OpenCLIP instead of CLIP yields lower retrieval accuracy on this dataset due to differences in training data curation (public datasets vs. carefully curated image-text correspondence).

## Evidence (verbatim from paper)

> To assess caption quality, we report standard captioning metrics including CLIPScore, CIDEr, BLEU-4 and METEOR, which measure fluency, descriptiveness, and semantic alignment with reference captions.

## Citation

```bibtex
@misc{nguyen2025openevents,
  title={OpenEvents V1: Large-Scale Benchmark Dataset for Multimodal Event Grounding},
  author={Nguyen et al. (2025)},
  year={2025},
  note={arXiv:2506.18372}
}
```

- arXiv: 2506.18372