# Ov Vsggen Eval

> Probes a model's ability to generate open-vocabulary video scene graphs by predicting objects, attributes, relations, and triplets from ground-truth trajectories. It evaluates semantic matching accuracy beyond exact label overlap and tests temporal grounding for dynamic relations. Use when the user wants to benchmark on PVSG, VidOR, VIPSeg, SVG2 test set, or asks about evaluating this task. Reports object/attribute/relation/triplet prediction accuracy (LLM-judged).

- Skill: `qhjqhj00/ov-vsggen-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/ov-vsggen-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/ov-vsggen-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/ov-vsggen-eval

---


# ov-vsggen-eval

> Synthetic Visual Genome 2: Extracting Large-scale Spatio-Temporal Scene Graphs from Videos — Gao et al. (2026) (arXiv:2602.23543, 2026)

## What this evaluates

Probes a model's ability to generate open-vocabulary video scene graphs by predicting objects, attributes, relations, and triplets from ground-truth trajectories. It evaluates semantic matching accuracy beyond exact label overlap and tests temporal grounding for dynamic relations.

## Datasets

- **PVSG** — total ?; splits: test (-1)
- **VidOR** — total ?; splits: test (-1)
- **VIPSeg** — total ?; splits: test (-1)
- **SVG2 test set** — total ?; splits: test (-1)

## Metrics

- `object/attribute/relation/triplet prediction accuracy (LLM-judged)` **(primary)** — range: percent
  - Proportion of predicted elements correctly matched to ground truth using an LLM judge. Matches are categorized as Identical, Synonym, Hypernym/Hyponym, Semantic Overlap, or Mismatch. Strict accuracy counts only Identical; lenient accuracy includes Synonym, Hypernym/Hyponym, and Semantic Overlap. Relation correctness additionally requires temporal grounding (IoU > 0.5 between predicted and ground-truth intervals). Triplet correctness requires a matched relation and lenient matches for both subject and object.

## Input / output format

**Input**: Ground-truth object trajectories, trajectory-aligned bounding boxes, and structured prompting templates.

**Output**: Structured scene-graph predictions following a JSON schema.

## Scoring recipe

```python
def evaluate(predictions, ground_truths):
    correct = 0
    total = len(predictions)
    for pred, gt in zip(predictions, ground_truths):
        match = llm_judge.compare(pred, gt)
        is_match = match in ["Identical", "Synonym", "Hypernym/Hyponym", "Semantic Overlap"]
        if not is_match:
            continue
        if pred.type == "relation":
            if IoU(pred.interval, gt.interval) > 0.5:
                correct += 1
        elif pred.type == "triplet":
            if llm_judge.lenient_match(pred.subject, gt.subject) and llm_judge.lenient_match(pred.object, gt.object):
                correct += 1
        else:
            correct += 1
    return (correct / total) * 100
```

## Common pitfalls

- Open-vocabulary generation makes traditional fixed-label metrics like Recall@k inadequate due to semantic variations.
- Benchmark annotation incompleteness inherently biases accuracy-based evaluation.
- Relation correctness requires temporal grounding (IoU > 0.5), not just label matching, and precision is omitted due to incomplete annotations.

## Evidence (verbatim from paper)

> We evaluate video scene graph generation in the open-vocabulary setting, assessing object, attribute, relation, and triplet prediction across multiple academic benchmarks and our fully annotated test set, and introduce an LLM-based judge to provide semantic, hierarchy-aware matching beyond exact label comparison. For object evaluation, we compute two accuracy levels: a strict score that considers predictions correct only if they fall into the identical category, and a lenient score that additionally accepts synonym, hypernym/hyponym, and semantic overlap. For relations, correctness additionally requires correct temporal grounding: the predicted relation interval must have IoU>0.5 with the ground-truth interval.

## Citation

```bibtex
@misc{gao2026svg2,
  title={Synthetic Visual Genome 2: Extracting Large-scale Spatio-Temporal Scene Graphs from Videos},
  author={Gao et al. (2026)},
  year={2026},
  note={arXiv:2602.23543}
}
```

- arXiv: 2602.23543