Helms Exploration Eval

helms-exploration-eval

HELM: Human-Preferred Exploration with Language Models — Liao et al. (2025) (arXiv:2503.07006, 2025)

What this evaluates

Evaluates the exploration efficiency and adaptability of a robot planner in unknown environments. It measures how well the system covers free space while minimizing travel distance and adapting to natural language preferences.

Datasets

• Simulated dungeon environments — total 100; splits: test (100)
• Indoor office environment (130m x 100m) — total 1; splits: test (1)

Metrics

• Travel Distance (primary) — range: other
  • Total Euclidean length of the robot's trajectory from start until >99% of the free area is covered.
• Replanning Time — range: other
  • Wall-clock time required for the planner to generate a new trajectory upon receiving a preference update or sensor input.

Input / output format

Input: Natural language prompt specifying task or human preferences, combined with real-time onboard sensor data (LiDAR) and partial map observations.

Output: Sequential robot motion commands forming an exploration trajectory.

Scoring recipe

def compute_travel_distance(trajectory, ground_truth_free_area):
    covered = 0.0
    total_dist = 0.0
    for i in range(1, len(trajectory)):
        total_dist += euclidean_dist(trajectory[i-1], trajectory[i])
        covered += newly_explored_area(trajectory[i])
        if covered / ground_truth_free_area > 0.99:
            return total_dist
    return total_dist

Common pitfalls

• Exploration completion is defined as >99% coverage, not 100%, to allow for sensor/model tolerances.
• The 'Optimal' baseline approximates the true optimum via TSP on sampled nodes covering ground-truth frontiers, not an exact solver.
• Results report mean and variance across 100 environments, so single-environment scores are not directly comparable to the paper's headline numbers.

Evidence (verbatim from paper)

We compare the travel distance of HELM to complete the exploration with several conventional planners... The mean and variance of the trajectory lengths required for full exploration are summarized in Table I. In our experiments, exploration is considered complete once more than 99% of the free area has been covered, allowing for minor tolerances in error.

Citation

@misc{liao2025helms,
  title={HELM: Human-Preferred Exploration with Language Models},
  author={Liao et al. (2025)},
  year={2025},
  note={arXiv:2503.07006}
}

• arXiv: 2503.07006