Ego Walk Nav Eval

ego-walk-nav-eval

EgoWalk: A Multimodal Dataset for Robot Navigation in the Wild — Akhtyanov et al. (2025) (arXiv:2505.21282, 2025)

What this evaluates

Evaluates the ability of visual navigation models to predict future robot trajectories from egocentric video frames and context history. It probes scale-invariant trajectory prediction and alignment with human navigation behavior under domain shift conditions.

Datasets

• EgoWalk — total ?; splits: test (1000)

Metrics

• MSE (primary) — range: [0, ∞)
  • Mean Squared Error between optimally scaled predicted and ground truth waypoints. Used to find the best scale factor before final evaluation.
• ADE — range: [0, ∞)
  • Average Displacement Error, computed as the mean L2 distance between each predicted waypoint and its ground truth counterpart across the trajectory.
• FDE — range: [0, ∞)
  • Final Displacement Error, computed as the L2 distance between the last predicted waypoint and the last ground truth waypoint.

Input / output format

Input: Current observation frame, N previous frames (context history), and ground truth 5 future waypoints.

Output: Predicted scale-free trajectory consisting of 5 future waypoints.

Scoring recipe

def evaluate_navigation(pred, gt):
    # pred and gt are arrays of shape (5, D) where D is spatial dimensions
    # 1. Find optimal scale s that minimizes MSE between s*pred and gt
    s = np.argmin([np.mean((s * pred - gt)**2) for s in np.linspace(0.1, 10, 1000)])
    scaled_pred = s * pred
    # 2. Compute metrics on scaled predictions
    mse = np.mean((scaled_pred - gt)**2)
    ade = np.mean(np.linalg.norm(scaled_pred - gt, axis=1))
    fde = np.linalg.norm(scaled_pred[-1] - gt[-1])
    return {'MSE': mse, 'ADE': ade, 'FDE': fde}

Common pitfalls

• Models output scale-free trajectories; direct MSE comparison without optimal scaling yields invalid results.
• Context window size (N) varies across models, making fair comparison difficult without standardization.
• Low trajectory prediction error does not guarantee successful navigation; direction and collision avoidance are often more critical in practice.

Evidence (verbatim from paper)

We sample around 1,000 test cases from our dataset trajectories. Each test case includes an observation frame (current observation), $N$ previous frames (context history, $N$ depends on the model), and five future waypoints (ground truth actions). We balance the test suite in terms of forward, left-turn and right-turn trajectories. Since the models generate scale-free trajectories, we follow common approach from the visual SLAM and first find the best scales using Mean Squared Error (MSE) criterion. The scaled trajectories are then evaluated using several metrics: MSE, Absolute Displacement Error (ADE) and Final Displacement Error (FDE).

Citation

@misc{akhtyanov2025egowalk,
  title={EgoWalk: A Multimodal Dataset for Robot Navigation in the Wild},
  author={Akhtyanov et al. (2025)},
  year={2025},
  note={arXiv:2505.21282}
}

• arXiv: 2505.21282