Ssg Generation Eval

ssg-generation-eval

Situational Scene Graph for Structured Human-centric Situation Understanding — Chinthani Sugandhika et al. (2024) (arXiv:2410.22829, 2024)

What this evaluates

Evaluates a model's ability to generate structured, human-centric scene graphs from images by jointly predicting verb predicates and fine-grained semantic role-value pairs for persons and objects. It probes multi-concurrent action understanding, affordance reasoning, and structured visual representation learning.

Datasets

• SSG dataset — total 25500; splits: test (-1); repo https://github.com/LUNAProject22/SSG
• Action Genome dataset — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: percent
  • Percentage of correctly predicted verb predicates under the PREDcls (with constraint) setting. For VQA subtasks, it is the percentage of correctly selected multiple-choice options.
• value — range: percent
  • Proportion of semantic roles where the model correctly predicts at least one role-value pair for that role.
• value-all — range: percent
  • Proportion of semantic roles where the model correctly predicts all associated role-value pairs for that role.
• value-two — range: percent
  • Proportion of semantic roles where the model correctly predicts at least two role-value pairs for that role.
• role-based accuracy — range: percent
  • Average of per-role exact-match accuracy scores across all semantic roles.

Input / output format

Input: RGB image (video frame). For VQA/reasoning subtasks, input includes the image plus a multiple-choice question (and optionally a text-formatted SSG graph).

Output: Structured prediction consisting of verb predicates and semantic role-value pairs for persons and objects. For VQA, a single selected option from multiple-choice options.

Scoring recipe

def score_ssg(preds, gold):
    # preds, gold: dicts mapping entity/role to predicted/gold values
    # 1. Verb Predicate Accuracy (PREDcls)
    verb_acc = sum(1 for p, g in zip(preds['verbs'], gold['verbs']) if p == g) / len(gold['verbs'])
    
    # 2. Value metrics per role
    role_values = {}
    for role in gold['roles']:
        g_vals = set(gold['roles'][role])
        p_vals = set(preds['roles'].get(role, []))
        correct_count = len(g_vals & p_vals)
        role_values[role] = correct_count
        
    # value: >=1 correct, value-two: >=2 correct, value-all: == all
    n_roles = len(role_values)
    val = sum(1 for c in role_values.values() if c >= 1) / n_roles
    val_two = sum(1 for c in role_values.values() if c >= 2) / n_roles
    val_all = sum(1 for c, total in zip(role_values.values(), gold['roles'].values()) if c == len(total)) / n_roles
    
    # role-based accuracy: avg of per-role exact match accuracy
    role_acc = sum(1 for c, total in zip(role_values.values(), gold['roles'].values()) if c == len(total)) / n_roles
    
    return {'verb_acc': verb_acc, 'value': val, 'value-two': val_two, 'value-all': val_all, 'role_based_acc': role_acc}

Common pitfalls

• The dataset contains only a single annotation per record, unlike prior works with multiple annotations, requiring adapted thresholding for value metrics.
• SRV classification uses a top-1 verb setting: if the predicted verb is incorrect, all associated semantic role-values are automatically marked incorrect.
• Object and person SRV classifications assume ground-truth bounding boxes are known (no object detection stage is performed).

Evidence (verbatim from paper)

For the verb predicate classification task, we use the metric accuracy (Acc) [[77]] under ”with constraint” predicate classification (PREDcls)[[27]] setting. Further, since semantic role-value classification is similar to the semantic role labelling in conventional verb-based situation frames, we adapt existing metrics value and value-all [[46]] along with a new metric called value-two. However, unlike in [[77], [46]], we have only single annotation per record. Therefore, we assess the value by determining whether the model accurately predicts at least one role-value for a given role out of all the roles associated with that particular semantic entity. The value-all metric evaluates if the model can accurately predict all the semantic role-values, while the value-two metric checks if at least two semantic role-values are correctly predicted out of all the roles.

Citation

@misc{sugandhika2024situational,
  title={Situational Scene Graph for Structured Human-centric Situation Understanding},
  author={Chinthani Sugandhika et al. (2024)},
  year={2024},
  note={arXiv:2410.22829}
}

• arXiv: 2410.22829