qhjqhj00 personalized-embodied-navigation-eval
PersONAL: Towards a Comprehensive Benchmark for Personalized Embodied Agents — Ziliotto et al. (2025) (arXiv:2509.19843, 2025)
What this evaluates
Evaluates an embodied agent's ability to navigate and ground objects based on user-specific ownership semantics provided only in text. It tests long-term memory, spatial reasoning, and the capacity to interpret personalized queries without relying on visual object cues.
Datasets
- PersONAL — total 2000; splits: test (-1); repo https://github.com/ZiliottoFilippoDev/PersONAL
Metrics
success_rate(primary) — range: [0, 1]- Fraction of episodes where the agent selects the STOP action while positioned within 1.0 meters of the target object.
Input / output format
Input: A personalized query (e.g., 'Find Lisa’s cabinet') and a detailed textual scene description specifying object ownership and spatial layout. For POG, these are provided after a pre-exploration phase.
Output: For PAN: a sequence of navigation actions (MOVE_FORWARD, TURN_LEFT/RIGHT, LOOK_UP/DOWN, STOP). For POG: coordinates on a pre-built spatial map.
Scoring recipe
def compute_success_rate(predictions, gold):
successful = 0
for pred, target in zip(predictions, gold):
if pred['action'] == 'STOP' and distance(pred['pos'], target['pos']) <= 1.0:
successful += 1
return successful / len(predictions)
Common pitfalls
- Confusing PAN (no prior knowledge, 500-step budget) with POG (allows 2,500-step pre-exploration and requires map coordinate output).
- Assuming image-based cues are used for ownership; the benchmark explicitly relies solely on textual descriptions.
- Ignoring strict step budget limits, which terminate episodes prematurely if exceeded.
Evidence (verbatim from paper)
An episode is considered successful if, after selecting the STOP action, the agent lies within 1.0 meters from the target object.
Citation
@misc{ziliotto2025personal,
title={PersONAL: Towards a Comprehensive Benchmark for Personalized Embodied Agents},
author={Ziliotto et al. (2025)},
year={2025},
note={arXiv:2509.19843}
}
- arXiv: 2509.19843