Csegg Eval

csegg-eval

Adaptive Visual Scene Understanding: Incremental Scene Graph Generation — Khandelwal et al. (2023) (arXiv:2310.01636, 2023)

What this evaluates

Evaluates continual learning capabilities in scene graph generation by measuring how models retain prior object-relationship knowledge while learning new tasks, handle long-tailed data distributions, and generalize to unseen objects and relationships across incremental learning scenarios.

Datasets

• CSEGG — total ?; splits: train (-1), test (-1); repo https://github.com/ZhangLab-DeepNeuroCogLab/CSEGG.git

Metrics

• Avg. R@20 (primary) — range: [0, 1]
  • Average recall@20 computed across all incremental tasks and object/relationship classes. Higher is better.
• F@20 — range: [0, 1]
  • F1-score@20 for relationship prediction averaged across tasks. Higher is better.
• FWT@20 — range: other
  • Forward Transfer@20 measuring performance gain on new tasks compared to a baseline trained from scratch.
• BWT@20 — range: other
  • Backward Transfer@20 measuring catastrophic forgetting by comparing final performance on old tasks to their peak performance.
• Gen Rbbox@20 — range: [0, 1]
  • Generalization recall@20 for detecting bounding boxes of unknown objects.
• Gen R@20 — range: [0, 1]
  • Generalization recall@20 for classifying known relationships among unknown objects.

Input / output format

Input: RGB images containing scenes with objects and relationships.

Output: Predicted scene graphs consisting of object bounding boxes, object class labels, and relationship triples (subject-predicate-object).

Scoring recipe

# Compute per-task metrics
for task in tasks:
    preds = model(images[task])
    golds = ground_truth[task]
    R20[task] = recall_at_k(preds.objects, golds.objects, k=20)
    F20[task] = f1_score(preds.rels, golds.rels, k=20)
# Aggregate continual learning metrics
avg_r20 = mean(R20)
f20 = mean(F20)
fwt20 = (R20[last] - R20[1]) / (num_tasks - 1)
bwt20 = mean(R20[last] - R20[initial])
# Generalization metrics (Scenario 3)
gen_rbbox20 = recall_at_k(preds.bboxes, unknown_gt.bboxes, k=20)
gen_r20 = recall_at_k(preds.rels, unknown_gt.rels, k=20)

Common pitfalls

• Avg.R@20 averages recall across all incremental tasks/classes, not just the current task, so it reflects retention rather than single-task performance.
• Long-tailed distributions cause severe tail-class forgetting; standard replay without sampling techniques (LVIS/BLS) underperforms significantly.
• Higher forgetting in replay methods can paradoxically improve generalization to unknown objects due to fixed output box limits, contradicting standard CL intuition.

Evidence (verbatim from paper)

We present Avg. R@20, F@20, FWT@20, and BWT@20 results for learning scenario 1 (S1) in Fig. 5(a)(b) and Fig.S11(a)(b). Notably, all continual learning baselines start from a similar Avg.R@20 in Task 1 and their performance drops over subsequent tasks. This implies that catastrophic forgetting about learned relationships occurs when the CSEGG models learn new relationships.

Citation

@misc{khandelwal2023csegg,
  title={Adaptive Visual Scene Understanding: Incremental Scene Graph Generation},
  author={Khandelwal et al. (2023)},
  year={2023},
  note={arXiv:2310.01636}
}

• arXiv: 2310.01636