Osmabench Eval

osmabench-eval

OSMa-Bench: Evaluating Open Semantic Mapping Under Varying Lighting Conditions — Popov et al. (2025) (arXiv:2503.10331, 2025)

What this evaluates

Evaluates open semantic mapping models' robustness to dynamic indoor lighting and motion conditions. It measures semantic segmentation accuracy and frequency-weighted IoU, alongside LLM-generated visual question answering accuracy on scene graphs.

Datasets

• ReplicaCAD — total 22; splits: test (-1)
• HM3D — total 8; splits: test (-1)

Metrics

• mAcc (primary) — range: [0, 1]
  • Mean accuracy computed from an overall confusion matrix (ReplicaCAD) or averaged per scene (HM3D), giving equal weight to all classes regardless of frequency.
• f-mIoU (primary) — range: [0, 1]
  • Frequency-weighted mean Intersection over Union, where voluminous objects like walls and floors contribute more to the score based on class frequency.
• VQA Accuracy — range: percent
  • Percentage of correctly answered questions generated by Gemini across categories such as existence, attributes, and spatial relations.

Input / output format

Input: RGB-D 3D scenes simulated in Habitat Sim with configurable lighting (baseline, nominal, camera, dynamic) and robot trajectories; scene graphs and LLM-generated questions for VQA.

Output: Semantic segmentation maps, scene graphs, and natural language answers to VQA questions.

Scoring recipe

# For ReplicaCAD (global confusion matrix)
conf_matrix = compute_confusion_matrix(predictions, ground_truth)
mAcc = mean(diag(conf_matrix) / conf_matrix.sum(axis=1))
f_mIoU = sum(class_freq * (diag(conf_matrix) / (conf_matrix.sum(axis=1) + conf_matrix.sum(axis=0) - diag(conf_matrix)))) / sum(class_freq)

# For HM3D (average per scene)
scene_mAcc = [compute_mAcc(scene_preds, scene_gt) for scene in scenes]
scene_f_mIoU = [compute_f_mIoU(scene_preds, scene_gt) for scene in scenes]
mAcc = mean(scene_mAcc)
f_mIoU = mean(scene_f_mIoU)

# VQA Accuracy
vqa_acc = sum(correct_answers) / total_questions * 100

Common pitfalls

• HM3D metrics are averaged per scene due to label inconsistencies across scenes, unlike ReplicaCAD which uses a global confusion matrix.
• Functional relationship questions were excluded from VQA evaluation due to LLM ambiguity.
• Class name formatting (underscores vs spaces) significantly impacts segmentation scores.
• Object merging in scene graphs inflates instance counts, skewing measurement-based questions.

Evidence (verbatim from paper)

Collecting mAcc and f-mIoU metrics from testing ConceptGraphs, BBQ, and OpenScene on scenes in ReplicaCAD and HM3D reveals key trade-offs. f-mIoU is weighted by class frequency, so voluminous objects like walls and floors matter more, while mAcc gives equal weight to all objects.

Citation

@misc{popov2025osmabench,
  title={OSMa-Bench: Evaluating Open Semantic Mapping Under Varying Lighting Conditions},
  author={Popov et al. (2025)},
  year={2025},
  note={arXiv:2503.10331}
}

• arXiv: 2503.10331