Egonormia Eval

egonormia-eval

EgoNormia: Benchmarking Physical Social Norm Understanding — Rezaei et al. (2025) (arXiv:2502.20490, 2025)

What this evaluates

Evaluates vision-language models' ability to understand and reason about physical-social norms in egocentric video scenarios. It probes whether models can correctly select normative actions, justify them, and identify plausible alternatives in conflict-prone situations.

Datasets

• EgoNormia — total 2053; splits: full (1853), verified (200)

Metrics

• Accuracy (primary) — range: percent
  • Proportion of correct predictions out of total instances. Calculated separately for action selection, justification, and combined (Both).
• IoU — range: [0, 1]
  • Intersection over Union between the set of predicted plausible actions and the set of ground-truth plausible actions. Used for the third subtask where multiple contextually-sensible choices exist.

Input / output format

Input: Three evaluation settings: (a) Blind: questions only; (b) Pipeline: rich text scene description (generated by Gemini 1.5 Flash) concatenated with questions; (c) Video: egocentric video sampled at 1 fps and concatenated left-to-right into a single image, plus questions. Chain-of-thought (CoT) prompting is applied to all non-reasoning models.

Output: Multiple-choice selection for the correct action and its justification, plus identification of plausible alternative actions. Models generate text responses following CoT prompting.

Scoring recipe

def compute_accuracy(preds, golds):
    correct = sum(1 for p, g in zip(preds, golds) if p == g)
    return correct / len(golds)

def compute_iou(preds, golds):
    iou_scores = []
    for p, g in zip(preds, golds):
        p_set, g_set = set(p), set(g)
        union = len(p_set | g_set)
        iou_scores.append(len(p_set & g_set) / union if union > 0 else 0.0)
    return sum(iou_scores) / len(iou_scores)

Common pitfalls

• Input modality heavily influences results; performance drops ~20-26% when visual input is removed (Blind setting).
• The third subtask uses IoU instead of Accuracy because multiple alternative actions are considered correct.
• CoT prompting is required for non-reasoning models to achieve comparable performance, which may inflate latency.

Evidence (verbatim from paper)

Accuracy is used in the first two subtasks with a single ground-truth answer; intersection over union (IoU) is used on the third subtask, where multiple contextually-sensible action choices exist.

Citation

@misc{rezaei2025egonormia,
  title={EgoNormia: Benchmarking Physical Social Norm Understanding},
  author={Rezaei et al. (2025)},
  year={2025},
  note={arXiv:2502.20490}
}

• arXiv: 2502.20490