qhjqhj00 thyme-scene-graph-eval
THYME: Temporal Hierarchical-Cyclic Interactivity Modeling for Video Scene Graphs in Aerial Footage — Trong-Thuan Nguyen et al. (2025) (arXiv:2507.09200, 2025)
What this evaluates
Evaluates a model's ability to generate dynamic video scene graphs by predicting inter-object relationships and attributes across multiple temporal frames. It specifically probes temporal consistency, handling of occlusions, and modeling of long-range dependencies in both ground-level and aerial video footage.
Datasets
- ASPIRe — total ?; splits: test (-1)
- AeroEye-v1.0 — total ?; splits: test (-1)
Metrics
R@20(primary) — range: percent- Recall@K measures the fraction of ground-truth predicates correctly predicted within the top-K ranked predictions. Evaluated at K=20, 50, and 100.
mR@20(primary) — range: percent- Mean Recall@K computes the average recall across all predicate classes to mitigate class imbalance. Evaluated at K=20, 50, and 100.
Input / output format
Input: Video sequences (ground-level or aerial) with annotated object bounding boxes, attributes, and ground-truth scene graph edges categorized into five interactivity types: Appearance, Situation, Position, Interaction, and Relation.
Output: Top-K predicted scene graph edges (subject-predicate-object triples) per video frame/clip, ranked by confidence score.
Scoring recipe
def compute_recall(preds, gold, k=20):
top_k = set(preds[:k])
correct = len(top_k.intersection(gold))
return (correct / len(gold)) * 100 if gold else 0
def compute_mean_recall(preds_list, gold_list, k=20):
recalls = [compute_recall(p, g, k) for p, g in zip(preds_list, gold_list)]
return sum(recalls) / len(recalls) if recalls else 0
Common pitfalls
- Evaluating only at top-20 without checking top-50/100 may overstate performance on long-tail predicate classes.
- Failing to account for temporal occlusions can cause models to incorrectly penalize valid predictions where objects temporarily disappear from the frame.
- Reporting only standard Recall without Mean Recall hides severe class imbalance issues in rare interactivity types like Interaction or Relation.
Evidence (verbatim from paper)
Following[[52], [32], [33]], we evaluate methods using Recall (R) and mean Recall (mR) at multiple thresholds, specifically, predictions from the top-20, top-50, and top-100.
Citation
@misc{nguyen2025thyme,
title={THYME: Temporal Hierarchical-Cyclic Interactivity Modeling for Video Scene Graphs in Aerial Footage},
author={Trong-Thuan Nguyen et al. (2025)},
year={2025},
note={arXiv:2507.09200}
}
- arXiv: 2507.09200