Cyclo Sgg Eval

cyclo-sgg-eval

CYCLO: Cyclic Graph Transformer Approach to Multi-Object Relationship Modeling in Aerial Videos — Nguyen et al. (2024) (arXiv:2406.01029, 2024)

What this evaluates

Evaluates a model's ability to generate scene graphs from aerial video sequences by predicting object relationships and interactions. It specifically probes long-range temporal dependency modeling and periodic interaction recognition in drone-captured footage across predicate classification, scene graph classification, and scene graph detection tasks.

Datasets

• AeroEye — total 2260; splits: train (1797), test (463)
• PVSG — total ?; splits: (unstated)
• ASPIRe — total ?; splits: (unstated)

Metrics

• Recall@K (R@K) — range: percent
  • The percentage of ground-truth predicates correctly predicted within the top K predictions for a video sequence.
• mean Recall@K (mR@K) (primary) — range: percent
  • The average of Recall@K across all predicate categories. This metric is used to handle long-tail class distributions common in scene graph generation.

Input / output format

Input: Video sequences of aerial/drone footage. Visual features are extracted using DINO (ResNet-50 backbone). For PredCls and SGCls, ground-truth object bounding boxes are provided; for SGDet, raw frames are processed.

Output: Scene graphs comprising predicted object classes, bounding boxes (SGDet only), and predicate relationships linking detected objects.

Scoring recipe

def compute_recall(preds, gold, k):
    top_k = preds[:k]
    return sum(1 for p in top_k if p in gold) / len(gold) * 100

def compute_mR(preds, gold, k):
    cat_correct, cat_total = {}, {}
    for p, g in zip(preds, gold):
        cat = g.category
        cat_total[cat] = cat_total.get(cat, 0) + 1
        if p in g: cat_correct[cat] = cat_correct.get(cat, 0) + 1
    recalls = [cat_correct[c]/cat_total[c] for c in cat_total]
    return sum(recalls) / len(recalls) * 100

Common pitfalls

• Failing to distinguish between PredCls (uses GT objects), SGCls (uses GT objects), and SGDet (requires object detection) evaluation protocols.
• Reporting raw Recall instead of mean Recall (mR), which masks performance on long-tail predicate categories.
• Using inconsistent K thresholds (20, 50, 100) without standardizing the reporting format across papers.

Evidence (verbatim from paper)

Evaluation Metrics. We evaluate models on two standard tasks in image-based scene graph generation followed by previous work[[74], [7]]* that are predicate classification (PredCls), scene graph classification (SGCls), and scene graph detection (SGDet). While SGCls predicts relationships given ground truth objects, SGDet involves detecting objects and predicting relationships. These tasks are evaluated using Recall (R@K) and mean Recall (mR@$K$), where $K\in{20,50,100}$.

Citation

@misc{nguyen2024cyclo,
  title={CYCLO: Cyclic Graph Transformer Approach to Multi-Object Relationship Modeling in Aerial Videos},
  author={Nguyen et al. (2024)},
  year={2024},
  note={arXiv:2406.01029}
}

• arXiv: 2406.01029