# Sigmacollab Eval

> Probes an AI system's ability to understand and assist in physically situated, goal-directed collaboration tasks using multimodal egocentric sensing. It evaluates real-time scene understanding, interaction modeling, and proactive guidance in fluid, human-AI collaborative scenarios. Use when the user wants to benchmark on SigmaCollab, or asks about evaluating this task. Reports classification.

- Skill: `qhjqhj00/sigmacollab-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/sigmacollab-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/sigmacollab-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Productivity
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/sigmacollab-eval

---


# sigmacollab-eval

> SigmaCollab: An Application-Driven Dataset for Physically Situated Collaboration — Bohus et al. (2025) (arXiv:2511.02560, 2025)

## What this evaluates

Probes an AI system's ability to understand and assist in physically situated, goal-directed collaboration tasks using multimodal egocentric sensing. It evaluates real-time scene understanding, interaction modeling, and proactive guidance in fluid, human-AI collaborative scenarios.

## Datasets

- **SigmaCollab** — total 85; splits: full (85); repo https://github.com/microsoft/SigmaCollab

## Metrics

- `classification` **(primary)** — range: [0, 1]
  - Accuracy of correctly assigning the taxonomy label (e.g., [Add Amount], [Invisible State], [Discrete Event], [Move], [Check]) to each sub-step in the task recipe.

## Input / output format

**Input**: Multimodal egocentric recordings (audio, video, depth, gaze, hand/head pose) synchronized with task instructions and sub-step taxonomy labels.

**Output**: Predicted taxonomy label for each sub-step, or a sequence of recognized actions/states aligned with the task recipe.

## Scoring recipe

```python
def score(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if p == g)
    return correct / len(gold) if gold else 0.0
```

## Common pitfalls

- Sub-step instructions contain implicit physical constraints (e.g., 'shake until cold to the touch') that are difficult to ground in static or short video clips.
- Taxonomy tags like [Invisible State] and [Check] require temporal reasoning and state inference beyond simple action recognition.
- Dataset captures untrained users collaborating with AI, so performance heavily depends on real-time interaction dynamics and user expertise.

## Evidence (verbatim from paper)

> The classification of the sub-step per the taxonomy described in Section [5.1] is also shown after each sub-step instructions, in bracketed light gray text.

## Citation

```bibtex
@misc{bohus2025sigmacollab,
  title={SigmaCollab: An Application-Driven Dataset for Physically Situated Collaboration},
  author={Bohus et al. (2025)},
  year={2025},
  note={arXiv:2511.02560}
}
```

- arXiv: 2511.02560