Hm3d Objectnav Eval

hm3d-objectnav-eval

Skip-SCAR: Hardware-Friendly High-Quality Embodied Visual Navigation — Liu et al. (2024) (arXiv:2405.14154, 2024)

What this evaluates

Evaluates an embodied AI agent's ability to navigate indoor 3D environments to find specific object categories using RGB-D observations. It measures both navigation quality (success and path efficiency) and computational efficiency (latency, memory, and skip ratio) on a large-scale dataset.

Datasets

• HabitatMatterport3D (HM3D) — total 4171566; splits: train (3971566), test (2000)

Metrics

• SPL (primary) — range: [0, 1]
  • Success weighted by Path Length. Computed as the average over episodes of (success_i * (oracle_shortest_path_length_i / actual_path_length_i)). Penalizes inefficient paths even if the goal is reached.
• SR — range: [0, 1]
  • Success Rate. Computed as the ratio of successful episodes to the total number of episodes.
• S-SPL — range: [0, 1]
  • Soft Success weighted by Path Length. A modified version of SPL that tracks the agent's progress towards the goal even when the episode fails.
• Skip Ratio — range: [0, 1]
  • The ratio of skipped semantic segmentation steps to the total steps for a navigation episode.
• GFLOPs — range: other
  • Giga floating point operations ($10^9$). Reports the amount of computation required per action step.

Input / output format

Input: RGB-D image observations at 640×480 resolution with 79° horizontal FOV, current semantic map/depth readings, and a target object category goal.

Output: Navigation actions (movement commands), semantic segmentation skip decisions, and object target probability predictions.

Scoring recipe

def compute_metrics(predictions, gold):
    total = len(predictions)
    successful = 0
    spl_sum = 0.0
    s_spl_sum = 0.0
    skip_ratios = []
    for pred, goal in zip(predictions, gold):
        dist_to_goal = pred.distance_to(goal)
        oracle_dist = goal.oracle_distance()
        actual_dist = pred.path_length()
        is_success = dist_to_goal < goal.threshold
        if is_success: successful += 1
        spl_sum += is_success * (oracle_dist / actual_dist)
        s_spl_sum += (oracle_dist - dist_to_goal) / oracle_dist
        skip_ratios.append(pred.skipped_steps / pred.total_steps)
    sr = successful / total
    spl = spl_sum / total
    s_spl = s_spl_sum / total
    skip_ratio = sum(skip_ratios) / total
    return {"SR": sr, "SPL": spl, "S-SPL": s_spl, "Skip Ratio": skip_ratio}

Common pitfalls

• SPL heavily penalizes inefficient paths even if the goal is reached, so a high SR does not guarantee a high SPL score.
• Hardware-dependent metrics (latency, memory, GFLOPs) vary drastically between GPU and CPU setups; comparisons must explicitly state the exact platform used.
• Naive step-skipping increases the skip ratio but degrades navigation quality; the adaptive skip predictor is required to maintain SPL while saving computation.

Evidence (verbatim from paper)

Metrics We use the following metrics for comparing the methods: Success Rate (SR) is the ratio of successful episodes. SPL (Success weighted by Path Length) is the success weighted by the length of the agent's path relative to the oracle shortest path length [10], [34]. SoftSPL (S-SPL) is a modified version of SPL that track agent's progress towards the goal, even when the episode fails [34]. Skip Ratio is the ratio for skipped semantic of the total steps for a navigation episode. GFLOPs ( $10^9$ floating point operations), reports amount of computations.

Citation

@misc{liu2024skipscar,
  title={Skip-SCAR: Hardware-Friendly High-Quality Embodied Visual Navigation},
  author={Liu et al. (2024)},
  year={2024},
  note={arXiv:2405.14154}
}

• arXiv: 2405.14154