Hm Eqa Eval

hm-eqa-eval

General-Purpose Robotic Navigation via LVLM-Orchestrated Perception, Reasoning, and Acting — Lange et al. (2025) (arXiv:2506.17462, 2025)

What this evaluates

This benchmark evaluates a robot's ability to autonomously explore an unseen indoor environment and answer multiple-choice questions requiring object identification, counting, spatial reasoning, and multi-goal navigation. It probes the agent's multimodal perception, iterative reasoning, and navigation efficiency in a dynamic, tool-invoking workflow.

Datasets

• HM-EQA — total ?; splits: test (-1)

Metrics

• Accuracy (primary) — range: percent
  • Percentage of multiple-choice questions answered correctly. Success, failure, and inconclusive outcomes are reported separately and always sum to 1.
• Mean Path Length — range: other
  • Average trajectory length in meters, measuring exploration efficiency.
• Mean Token Usage — range: other
  • Average number of tokens processed by the LVLM, serving as a proxy for computational cost and runtime.

Input / output format

Input: Multimodal observations from the Habitat Lab simulator (e.g., RGB-D frames, scene graphs, occupancy grids) and a multiple-choice navigation query. The agent iteratively queries perception modules and reasons over these inputs.

Output: Sequential navigation actions (trajectory steps) and a final selected multiple-choice answer.

Scoring recipe

def compute_accuracy(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if p == g)
    return correct / len(gold) * 100

def compute_path_length(trajectories):
    return sum(len(t) for t in trajectories) / len(trajectories)

def compute_token_usage(token_counts):
    return sum(token_counts) / len(token_counts)

Common pitfalls

• Inconclusive runs (exceeding 500 reasoning steps or $5 budget) are reported separately and do not sum to 100% with success/failure rates.
• Token usage and path length are highly variable; comparing them across methods requires accounting for different computational budgets and exploration strategies.
• Accuracy is calculated only on completed tasks, but inconclusive runs are excluded from the success/failure ratio, which can skew perceived performance.

Evidence (verbatim from paper)

We benchmark using three metrics: (1) Accuracy, the percentage of multiple-choice questions answered correctly; (2) Mean Path Length, the average trajectory length (in meters), measuring exploration efficiency; and (3) Mean Token Usage, the average number of tokens processed by the LVLM, which serves as a proxy for computational cost and runtime.

Citation

@misc{lange2025arnanavigation,
  title={General-Purpose Robotic Navigation via LVLM-Orchestrated Perception, Reasoning, and Acting},
  author={Lange et al. (2025)},
  year={2025},
  note={arXiv:2506.17462}
}

• arXiv: 2506.17462