Saw Bench Eval

saw-bench-eval

Learning Situated Awareness in the Real World — Li et al. (2026) (arXiv:2602.16682, 2026)

What this evaluates

Evaluates multimodal foundation models' ability to perform observer-centric spatial reasoning, path integration, and camera geometry inference using egocentric videos recorded from smart glasses. It probes sustained tracking of intermediate movements and the distinction between physical translation and camera rotation.

Datasets

• SAW-BENCH — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: [0, 1]
  • Percentage of correctly answered multiple-choice questions after extracting the predicted answer via a regular-expression parser, with a GPT-4o-mini fallback for failed extractions.

Input / output format

Input: Egocentric video clips from smart glasses paired with a multiple-choice question about spatial reasoning, trajectory shape, or memory.

Output: Free-form text response containing the selected option, which is then parsed via regex or GPT-4o-mini to extract the final answer.

Scoring recipe

def compute_accuracy(predictions, golds):
    correct = 0
    for pred_raw, gold in zip(predictions, golds):
        pred = extract_answer(pred_raw) # regex parser, fallback to GPT-4o-mini
        if pred == gold:
            correct += 1
    return correct / len(golds)

Common pitfalls

• Models frequently conflate camera rotation/panning with the observer's physical translation.
• Open-source models often rely on shortcut cues from key frames (e.g., first and last) instead of tracking full trajectory dynamics.
• Static video captions (Socratic baseline) discard critical observer-centric cues like viewpoint changes and temporal structure.

Evidence (verbatim from paper)

We use accuracy as the primary metric to evaluate model performance on SAW-BENCH. Following recent benchmarks for foundation model evaluation (Wang et al., 2024b, Shangguan et al., 2025, Zhao et al., 2025), we first apply a regular-expression-based parser to extract the predicted answer from each model's raw response. If the parser fails, we additionally use GPT-4o-mini to extract the answer from the raw output. Prompt used for answer extraction is provided in §B.3.

Citation

@misc{li2026learning,
  title={Learning Situated Awareness in the Real World},
  author={Li et al. (2026)},
  year={2026},
  note={arXiv:2602.16682}
}

• arXiv: 2602.16682