Hyperglm Eval

hyperglm-eval

HyperGLM: HyperGraph for Video Scene Graph Generation and Anticipation — Trong-Thuan Nguyen et al. (2024) (arXiv:2411.18042, 2024)

What this evaluates

Evaluates a multimodal model's ability to generate and anticipate scene graphs from video frames, capturing spatial object relationships and causal temporal transitions. It also tests video question answering, captioning, and relation reasoning capabilities by leveraging hypergraph structures to model multi-way interactions.

Datasets

• VSGR — total ?; splits: test (-1)
• PVSG — total ?; splits: test (-1)
• Action Genome — total ?; splits: test (-1)

Metrics

• Recall (R) / mean Recall (mR) (primary) — range: percent
  • Fraction of ground-truth object-relation triples correctly predicted in the top-K ranked predictions. mR averages Recall across all predicate classes to mitigate long-tail bias.
• Accuracy — range: percent
  • Proportion of correctly answered questions or correctly inferred relations out of the total number of instances.
• Precision — range: percent
  • Proportion of predicted triples/answers that are correct out of all predictions made.
• F1 Score — range: percent
  • Harmonic mean of Precision and Recall: 2 * (Precision * Recall) / (Precision + Recall).
• CIDEr — range: percent
  • TF-IDF weighted n-gram similarity between generated captions and reference captions, emphasizing rare but informative words.
• ROUGE-L — range: percent
  • F1 score based on the longest common subsequence between generated and reference text.
• BLEU-4 — range: percent
  • Geometric mean of modified precision scores for 1- to 4-grams, with brevity penalty.
• MENTOR — range: percent
  • Metric for video captioning that evaluates temporal consistency and object tracking coherence in generated descriptions.

Input / output format

Input: Raw video frames (encoded into tokens via CLIP-ViT-L-336) and, for VQA/VC/RR tasks, corresponding text prompts or questions.

Output: For SGG/SGA: ranked lists of object-relation triples (scene graphs). For VQA/RR: text answers. For VC: generated video captions.

Scoring recipe

def calc_recall(pred_triples, gold_triples, k):
    top_k = pred_triples[:k]
    return sum(1 for t in gold_triples if t in top_k) / len(gold_triples)

def calc_acc_f1(preds, golds):
    correct = sum(p == g for p, g in zip(preds, golds))
    acc = correct / len(golds)
    tp = sum(1 for p, g in zip(preds, golds) if p == g)
    fp = sum(1 for p, g in zip(preds, golds) if p != g)
    fn = sum(1 for p, g in zip(preds, golds) if p != g)
    prec = tp / (tp + fp) if (tp + fp) > 0 else 0
    rec = tp / (tp + fn) if (tp + fn) > 0 else 0
    f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
    return acc, prec, rec, f1

Common pitfalls

• Recall thresholds vary across datasets (e.g., @10, @20, @50 for SGA; @20, @50, @100 for SGG), so results are not directly comparable without checking the K value.
• Mean Recall (mR) is used to handle long-tail predicate distributions, but it can mask poor performance on frequent classes if not reported alongside standard Recall.
• Video input fraction (F) significantly impacts SGA performance; models trained or evaluated at F=0.9 may not generalize to lower observation ratios without explicit testing.

Evidence (verbatim from paper)

We evaluate the SGG and SGA tasks using the Recall and mean Recall scores. In addition, we evaluate the VQA and RR tasks by Accuracy, Precision, Recall, and F1 scores. For the VC task, we utilize CIDEr, MENTOR, ROUGE-L, and BLEU-4 scores to validate our performance.

Citation

@misc{nguyen2024hyperglm,
  title={HyperGLM: HyperGraph for Video Scene Graph Generation and Anticipation},
  author={Trong-Thuan Nguyen et al. (2024)},
  year={2024},
  note={arXiv:2411.18042}
}

• arXiv: 2411.18042