qhjqhj00 physicalai-warehouse-eval
SmolRGPT: Efficient Spatial Reasoning for Warehouse Environments with 600M Parameters — Traore et al. (2025) (arXiv:2509.15490, 2025)
What this evaluates
Evaluates a model's capacity for metric spatial reasoning, object enumeration, relative spatial comparisons, and topological/directional relationship understanding within real-world warehouse environments.
Datasets
- PhysicalAI-Spatial-Intelligence-Warehouse — total ?; splits: train (-1), val (-1), test (-1)
Metrics
normalized exact-match accuracy(primary) — range: [0, 1]- Accuracy computed by comparing model predictions against ground-truth answers after applying a normalization protocol that accommodates common variations in numeric and word representations.
Input / output format
Input: A real-world warehouse scene image, a natural language question (distance, count, multiple-choice grounding, or spatial relation query), and region-indexed objects with a unified coordinate system.
Output: A normalized answer string (numeric value or word representation) corresponding to the question type.
Scoring recipe
def score(predictions, golds):
correct = 0
for pred, gold in zip(predictions, golds):
pred_norm = normalize_answer(pred)
gold_norm = normalize_answer(gold)
if pred_norm == gold_norm:
correct += 1
return correct / len(predictions)
Common pitfalls
- Failing to apply the dataset's normalization protocol before comparison, leading to false negatives for semantically identical answers (e.g., '5m' vs '5 meters').
- Misinterpreting relative spatial references without leveraging the provided region coordinate system.
- Applying a single scoring threshold across all four question categories instead of using category-specific normalization rules.
Evidence (verbatim from paper)
To support consistent assessment, the dataset offers standardized training, validation, and test splits, with normalized answer formats. Each instance includes objects indexed by region within the warehouse, allowing for precise spatial referencing via a unified coordinate system. The normalization protocol accommodates common variations in response formats, supporting a range of numeric and word representations while maintaining consistency across different types of queries.
Citation
@misc{traore2025smolrgpt,
title={SmolRGPT: Efficient Spatial Reasoning for Warehouse Environments with 600M Parameters},
author={Traore et al. (2025)},
year={2025},
note={arXiv:2509.15490}
}
- arXiv: 2509.15490