Ov Vsggen Eval

ov-vsggen-eval

Synthetic Visual Genome 2: Extracting Large-scale Spatio-Temporal Scene Graphs from Videos — Gao et al. (2026) (arXiv:2602.23543, 2026)

What this evaluates

Probes a model's ability to generate open-vocabulary video scene graphs by predicting objects, attributes, relations, and triplets from ground-truth trajectories. It evaluates semantic matching accuracy beyond exact label overlap and tests temporal grounding for dynamic relations.

Datasets

• PVSG — total ?; splits: test (-1)
• VidOR — total ?; splits: test (-1)
• VIPSeg — total ?; splits: test (-1)
• SVG2 test set — total ?; splits: test (-1)

Metrics

• object/attribute/relation/triplet prediction accuracy (LLM-judged) (primary) — range: percent
  • Proportion of predicted elements correctly matched to ground truth using an LLM judge. Matches are categorized as Identical, Synonym, Hypernym/Hyponym, Semantic Overlap, or Mismatch. Strict accuracy counts only Identical; lenient accuracy includes Synonym, Hypernym/Hyponym, and Semantic Overlap. Relation correctness additionally requires temporal grounding (IoU > 0.5 between predicted and ground-truth intervals). Triplet correctness requires a matched relation and lenient matches for both subject and object.

Input / output format

Input: Ground-truth object trajectories, trajectory-aligned bounding boxes, and structured prompting templates.

Output: Structured scene-graph predictions following a JSON schema.

Scoring recipe

def evaluate(predictions, ground_truths):
    correct = 0
    total = len(predictions)
    for pred, gt in zip(predictions, ground_truths):
        match = llm_judge.compare(pred, gt)
        is_match = match in ["Identical", "Synonym", "Hypernym/Hyponym", "Semantic Overlap"]
        if not is_match:
            continue
        if pred.type == "relation":
            if IoU(pred.interval, gt.interval) > 0.5:
                correct += 1
        elif pred.type == "triplet":
            if llm_judge.lenient_match(pred.subject, gt.subject) and llm_judge.lenient_match(pred.object, gt.object):
                correct += 1
        else:
            correct += 1
    return (correct / total) * 100

Common pitfalls

• Open-vocabulary generation makes traditional fixed-label metrics like Recall@k inadequate due to semantic variations.
• Benchmark annotation incompleteness inherently biases accuracy-based evaluation.
• Relation correctness requires temporal grounding (IoU > 0.5), not just label matching, and precision is omitted due to incomplete annotations.

Evidence (verbatim from paper)

We evaluate video scene graph generation in the open-vocabulary setting, assessing object, attribute, relation, and triplet prediction across multiple academic benchmarks and our fully annotated test set, and introduce an LLM-based judge to provide semantic, hierarchy-aware matching beyond exact label comparison. For object evaluation, we compute two accuracy levels: a strict score that considers predictions correct only if they fall into the identical category, and a lenient score that additionally accepts synonym, hypernym/hyponym, and semantic overlap. For relations, correctness additionally requires correct temporal grounding: the predicted relation interval must have IoU>0.5 with the ground-truth interval.

Citation

@misc{gao2026svg2,
  title={Synthetic Visual Genome 2: Extracting Large-scale Spatio-Temporal Scene Graphs from Videos},
  author={Gao et al. (2026)},
  year={2026},
  note={arXiv:2602.23543}
}

• arXiv: 2602.23543