Hsri Social Reasoning Eval

hsri-social-reasoning-eval

The Human Robot Social Interaction (HSRI) Dataset: Benchmarking Foundational Models' Social Reasoning — Lee et al. (2025) (arXiv:2504.13898, 2025)

What this evaluates

Evaluates foundational models' ability to detect social errors and competencies, identify specific social attributes, reason about sequential interaction flow (pre/post conditions), and generate rationales and corrective actions in human-robot interaction scenarios.

Datasets

• HSRI — total 440; splits: test (-1)

Metrics

• accuracy (primary) — range: [0, 1]
  • The proportion of instances where the model's extracted answer exactly matches or contains the ground-truth answer option. Used across all eight tasks.
• macro-F1 — range: [0, 1]
  • The unweighted average of F1 scores computed per class. Used for multi-class and multi-label classification tasks to account for class imbalance.
• Partial Match (PM) — range: [0, 1]
  • The proportion of instances where the model correctly predicts at least one of the true multi-label attributes, used specifically for the social attribute identification task.

Input / output format

Input: Contextual information consisting of a transcript and either 15 video frames (for image-based models) or raw video (for video-based models), combined with a task-specific question.

Output: A model-generated token sequence O, which is post-processed using an LLM with Pydantic to coerce the output into the expected structured answer options.

Scoring recipe

def compute_score(predictions, gold):
    correct = 0
    for pred, g in zip(predictions, gold):
        # Post-process pred to extract answer option if needed
        # Check if gold answer is a contiguous subsequence of pred
        if g in pred:
            correct += 1
    return correct / len(predictions)

Common pitfalls

• The correctness function checks for contiguous subsequence containment (A ⊆ O) rather than exact string matching, which can yield false positives if the model outputs conversational filler containing the answer.
• Post-processing requires an external LLM with Pydantic to extract answers, introducing latency and potential extraction failures that are not captured by the raw model output.
• Partial Match (PM) only requires one correct label in multi-label tasks, which can inflate performance metrics when the attribute space is large or imbalanced.

Evidence (verbatim from paper)

For the task of social attribute identification, we evaluate with accuracy and macro-F1 (F1) scores. As there can be more than a single attribute label associated with a sample, we further report Partial Match (PM) to evaluate the proportion of instances where the model correctly predicts at least one of the true labels.

Citation

@misc{lee2025hsri,
  title={The Human Robot Social Interaction (HSRI) Dataset: Benchmarking Foundational Models' Social Reasoning},
  author={Lee et al. (2025)},
  year={2025},
  note={arXiv:2504.13898}
}

• arXiv: 2504.13898