Embodied AI Objectnav Eval

embodied-ai-objectnav-eval

Selective Visual Representations Improve Convergence and Generalization for Embodied AI — Eftekhar et al. (2023) (arXiv:2311.04193, 2023)

What this evaluates

Evaluates embodied AI agents' ability to navigate to target objects in 3D environments and perform manipulation tasks. It probes spatial reasoning, path efficiency, trajectory smoothness, and zero-shot generalization across different simulation domains and visual styles.

Datasets

• ProcTHOR-10k — total ?; splits: train (-1), test (-1)
• ArchitecTHOR — total ?; splits: test (-1)
• AI2-iTHOR — total ?; splits: test (-1)
• RoboTHOR — total ?; splits: test (-1)
• ManipulaTHOR — total ?; splits: train (-1), val (-1), test (-1)
• Habitat 2022 ObjectNav — total ?; splits: test (-1)

Metrics

• SR (Success Rate) (primary) — range: percent
  • Percentage of episodes where the agent successfully reaches the target object. Calculated as the average of binary success indicators across all episodes.
• SPL (Success Weighted by Path Length) — range: [0, 1]
  • Measures path distance efficiency: 1/N * sum(S_i * min(l_i, p_i) / max(l_i, p_i)), where l_i is shortest path distance, p_i is traveled distance, and S_i is success indicator.
• SEL (Success Weighted by Episode Length) — range: [0, 1]
  • Measures step/time efficiency: 1/N * sum(S_i * min(w_i, e_i) / max(w_i, e_i)), where w_i is shortest episode length, e_i is actual episode length, and S_i is success indicator.
• Curvature — range: other
  • Quantifies trajectory smoothness using differential geometry: k = (dxddy - dyddx) / (dx^2 + dy^2)^(3/2) for each coordinate, averaged across all time steps. Higher values indicate more rotations.
• EL (Episode Length) — range: other
  • Total number of steps or time steps taken by the agent to complete an episode.
• PU (PickUp Success Rate) — range: percent
  • Percentage of episodes where the agent successfully picks up the source object in manipulation tasks.
• Invalid Actions — range: percent
  • Percentage of actions taken by the agent that are invalid within the environment constraints.

Input / output format

Input: RGB images (or RGB-D + segmentation masks for manipulation) encoded via a frozen CLIP ResNet-50, compressed by a 2-layer CNN, then concatenated with a 32-dim goal embedding and previous action embedding. The combined tensor is passed through a 256x10 codebook bottleneck.

Output: Action probability distribution and state value estimation from an actor-critic network, derived from a recurrent state encoder processing the codebook-compressed embedding.

Scoring recipe

def compute_metrics(predictions, gold):
    sr = sum(1 for p, g in zip(predictions, gold) if p.success) / len(predictions)
    spl = sum(p.success * min(p.path_len, g.shortest_path) / max(p.path_len, g.shortest_path) for p, g in zip(predictions, gold)) / len(predictions)
    sel = sum(p.success * min(p.episode_len, g.shortest_episode_len) / max(p.episode_len, g.shortest_episode_len) for p, g in zip(predictions, gold)) / len(predictions)
    curvature = sum(k for k in predictions.trajectory_curvatures) / len(predictions.trajectory_curvatures)
    return {'SR': sr, 'SPL': spl, 'SEL': sel, 'Curvature': curvature}

Common pitfalls

• SPL measures distance efficiency, not time/energy or rotation cost, which can mislead claims about agent efficiency.
• Curvature quantifies trajectory smoothness; higher values indicate more rotations/changes in direction, not necessarily task failure.
• Zero-shot evaluation assumes no fine-tuning on target benchmarks, relying solely on ProcTHOR-10k pretraining.

Evidence (verbatim from paper)

SPL is defined as $\frac{1}{N}\sum_{i=1}^{N}S_{i}\frac{l_{i}}{max(l_{i},p_{i})}$ where $l_{i}$ is the shortest possible path (traveled distance) to the target object, $p_{i}$ is the taken path by the agent and $S_{i}$ is a binary indicator of success for episode $i$. As shown in the table, EmbCLIP-Codebook outperforms in Success Rate and solves the task with fewer steps (smaller EL). However, it lags behind in the SPL metric. This discrepancy arises because SPL is evaluated based on the distance traveled rather than the actual number of steps taken. This highlights a limitation in this metric since the efficiency of a path should consider factors like time and energy consumption. Actions such as rotations and look-ups/downs, which also consume time and energy, are not accounted for in this metric. In light of this observation, we further report Success Weighted by Episode Length (SEL): $\frac{1}{N}\sum_{i=1}^{N}S_{i}\frac{w_{i}}{max(w_{i},e_{i})}$, where $w_{i}$ is the shortest possible episode length to the target object, and $e_{i}$ is the episode length produced by the agent.

Citation

@misc{eftekhar2023selective,
  title={Selective Visual Representations Improve Convergence and Generalization for Embodied AI},
  author={Eftekhar et al. (2023)},
  year={2023},
  note={arXiv:2311.04193}
}

• arXiv: 2311.04193