Spamem Eval

spamem-eval

SpaMEM: Benchmarking Dynamic Spatial Reasoning via Perception-Memory Integration in Embodied Environments — Liao et al. (2026) (arXiv:2604.22409, 2026)

What this evaluates

This benchmark evaluates dynamic spatial reasoning and perception-memory integration in embodied environments. It probes models across three levels: static spatial perception, text-conditioned temporal memory, and visual-conditioned temporal memory, testing capabilities like object recognition, visual grounding, depth estimation, trajectory tracking, and long-horizon state reconstruction.

Datasets

• SpaMEM — total ?; splits: test (-1)

Metrics

• mIoU (primary) — range: [0, 1]
  • Mean Intersection over Union between predicted and ground-truth bounding boxes for visual grounding.
• F1 — range: [0, 1]
  • Harmonic mean of precision and recall for semantic object recognition.
• AbsRel — range: other
  • Mean absolute relative error for depth and proximity estimation.
• Acc@1 — range: [0, 1]
  • Accuracy of predicting the exact object count.
• TL — range: [0, 1]
  • Accuracy of temporal localization (identifying correct frames or timesteps).
• STT — range: [0, 1]
  • Accuracy of spatio-temporal tracking (reconstructing object trajectories across containers).
• SED — range: [0, 1]
  • Accuracy of short-term event detection for add and remove operations.
• CSR — range: [0, 1]
  • Accuracy of cumulative state reconstruction (global inventory update at episode end).

Input / output format

Input: Egocentric RGB frames, optionally paired with aligned depth maps (RGB-D). For Level 2, explicit text-based state histories are provided. Queries are issued either step-by-step (short-term) or at the end of an episode (long-term).

Output: Model must produce task-specific predictions: bounding box coordinates (grounding), object class labels (recognition), depth values (estimation), relative directions (left/right/behind), integer counts, temporal frame indices (localization), trajectory paths (tracking), and updated state/inventory lists (event detection & reconstruction).

Scoring recipe

def score(predictions, gold):
    scores = {}
    scores['mIoU'] = mean([iou(p, g) for p, g in zip(predictions['boxes'], gold['boxes'])])
    tp = sum(1 for p, g in zip(predictions['labels'], gold['labels']) if p == g)
    scores['F1'] = 2 * tp / (len(predictions['labels']) + len(gold['labels']) + 1e-8)
    scores['AbsRel'] = mean([abs(p - g) / (g + 1e-8) for p, g in zip(predictions['depth'], gold['depth'])])
    scores['Acc@1'] = mean([1 if p == g else 0 for p, g in zip(predictions['count'], gold['count'])])
    scores['TL'] = mean([1 if p == g else 0 for p, g in zip(predictions['timestep'], gold['timestep'])])
    scores['STT'] = mean([1 if p == g else 0 for p, g in zip(predictions['trajectory'], gold['trajectory'])])
    scores['SED'] = mean([1 if p == g else 0 for p, g in zip(predictions['event'], gold['event'])])
    scores['CSR'] = mean([1 if p == g else 0 for p, g in zip(predictions['state'], gold['state'])])
    return scores

Common pitfalls

• Text-conditioned (L2) performance overestimates true visual memory; models excel at bookkeeping with symbolic history but collapse without it.
• Semantic recognition accuracy does not transfer to spatial grounding; high F1 often coexists with near-zero mIoU.
• Short-term event detection success does not compose into long-term state reconstruction; local change signals fail to update global inventory.

Evidence (verbatim from paper)

Visual grounding (VGL), measured by mean IoU (mIoU), is nearly non-functional: scores remain at 0.00–0.01 for almost all models. This indicates that while models may recognize object presence, they struggle to map semantics to coordinate-consistent localization in cluttered indoor scenes.

Citation

@misc{liao2026spamem,
  title={SpaMEM: Benchmarking Dynamic Spatial Reasoning via Perception-Memory Integration in Embodied Environments},
  author={Liao et al. (2026)},
  year={2026},
  note={arXiv:2604.22409}
}

• arXiv: 2604.22409