Unrealzoo Eval

unrealzoo-eval

UnrealZoo: Enriching Photo-realistic Virtual Worlds for Embodied AI — Zhong et al. (2024) (arXiv:2412.20977, 2024)

What this evaluates

Evaluates embodied AI agents' capabilities in complex, photo-realistic 3D open-world environments. Specifically probes visual navigation on unstructured terrain, active visual tracking across diverse scenes, and social tracking under dynamic distractions, varying morphologies, and different control frequencies.

Datasets

• UnrealZoo — total 100; splits: test (-1); repo https://github.com/UnrealZoo/unrealzoo-gym

Metrics

• Success Rate (SR) (primary) — range: [0, 1]
  • Percentage of episodes where the agent successfully reaches the target or completes the required steps (e.g., 500 steps) out of 50 total episodes.
• Average Episode Length (EL) — range: other
  • Average number of steps per episode over 50 episodes.
• Success weighted by Path Length (SPL) — range: [0, 1]
  • Standard metric that accounts for both success and path efficiency, calculated as the average of the success indicator multiplied by the ratio of the shortest possible path to the actual path length across episodes.
• Average Episodic Return (ER) — range: other
  • Mean episodic return over 50 episodes, reflecting overall tracking performance and reward accumulation.

Input / output format

Input: First-person view images (or segmentation masks) and relative position to target; for tracking tasks, raw observed images and target bounding box coordinates.

Output: Discrete navigation actions (e.g., move forward/backward, turn left/right, jump, crouch) or continuous velocity commands mapped to predefined action spaces.

Scoring recipe

def compute_metrics(predictions, gold, total_episodes=50):
    sr = sum(1 for p, g in zip(predictions, gold) if p == g) / total_episodes
    el = sum(len(p) for p in predictions) / total_episodes
    spl = sum((1 if p == g else 0) * (shortest_path / len(p)) for p, g in zip(predictions, gold)) / total_episodes
    er = sum(p.reward for p in predictions) / total_episodes
    return {'SR': sr, 'EL': el, 'SPL': spl, 'ER': er}

Common pitfalls

• Unstructured terrain requires diverse locomotion (climbing, jumping) rather than standard grid movement, making path efficiency (SPL) harder to compute and interpret compared to traditional benchmarks.
• Control frequency latency significantly impacts performance (drops below 10 FPS), but this condition is often omitted in standard embodied benchmarks, leading to over-optimistic results.
• Cross-embodiment generalization tests transfer from human to quadrupedal morphologies, which drastically reduces success rates and is frequently overlooked in policy evaluation.

Evidence (verbatim from paper)

Evaluation Metrics. We employ two key metrics to evaluate visual navigation agents: 1) Average Episode Length (EL), representing the average number of steps per episode over 50 episodes. 2) Success Rate (SR), measuring the percentage of episodes the agent successfully navigates to the target object out of 50 total episodes, which represents the navigation capability in the wild environment. 3) Success weighted by Path Length (SPL), a standard metric that accounts for both success and path efficiency.

Citation

@misc{zhong2024unrealzoo,
  title={UnrealZoo: Enriching Photo-realistic Virtual Worlds for Embodied AI},
  author={Zhong et al. (2024)},
  year={2024},
  note={arXiv:2412.20977}
}

• arXiv: 2412.20977