Sigmacollab Eval

sigmacollab-eval

SigmaCollab: An Application-Driven Dataset for Physically Situated Collaboration — Bohus et al. (2025) (arXiv:2511.02560, 2025)

What this evaluates

Probes an AI system's ability to understand and assist in physically situated, goal-directed collaboration tasks using multimodal egocentric sensing. It evaluates real-time scene understanding, interaction modeling, and proactive guidance in fluid, human-AI collaborative scenarios.

Datasets

• SigmaCollab — total 85; splits: full (85); repo https://github.com/microsoft/SigmaCollab

Metrics

• classification (primary) — range: [0, 1]
  • Accuracy of correctly assigning the taxonomy label (e.g., [Add Amount], [Invisible State], [Discrete Event], [Move], [Check]) to each sub-step in the task recipe.

Input / output format

Input: Multimodal egocentric recordings (audio, video, depth, gaze, hand/head pose) synchronized with task instructions and sub-step taxonomy labels.

Output: Predicted taxonomy label for each sub-step, or a sequence of recognized actions/states aligned with the task recipe.

Scoring recipe

def score(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if p == g)
    return correct / len(gold) if gold else 0.0

Common pitfalls

• Sub-step instructions contain implicit physical constraints (e.g., 'shake until cold to the touch') that are difficult to ground in static or short video clips.
• Taxonomy tags like [Invisible State] and [Check] require temporal reasoning and state inference beyond simple action recognition.
• Dataset captures untrained users collaborating with AI, so performance heavily depends on real-time interaction dynamics and user expertise.

Evidence (verbatim from paper)

The classification of the sub-step per the taxonomy described in Section [5.1] is also shown after each sub-step instructions, in bracketed light gray text.

Citation

@misc{bohus2025sigmacollab,
  title={SigmaCollab: An Application-Driven Dataset for Physically Situated Collaboration},
  author={Bohus et al. (2025)},
  year={2025},
  note={arXiv:2511.02560}
}

• arXiv: 2511.02560