Mtc Locomotion Eval

mtc-locomotion-eval

Moving Through Clutter: Scaling Data Collection and Benchmarking for 3D Scene-Aware Humanoid Locomotion via Virtual Reality — Wang et al. (2026) (arXiv:2603.05993, 2026)

What this evaluates

Probes a humanoid robot's ability to navigate procedurally generated 3D cluttered environments while adapting full-body kinematics to geometric constraints. It quantifies how much a policy deviates from nominal flat-ground walking and measures collision safety against complex scene geometry.

Datasets

• MTC Dataset — total 348; splits: full (348)

Metrics

• Motion Adaptation Score (primary) — range: other
  • Computed as the Fréchet distance between the empirical feature distribution of a test trajectory and a reference flat-ground walking distribution across four kinematic subspaces (posture, vertical motion, foot interaction, smoothness). Subspace distances are normalized and uniformly weighted to produce a scalar.
• Collision Frequency ($R_{\text{col}}$) — range: [0, 1]
  • Fraction of trajectory frames where penetration depth exceeds zero: $R_{\text{col}} = \frac{1}{T}\sum_{t=1}^{T}\mathbf{1}[d_{t}>0]$.
• Max Penetration Depth ($d_{\max}$) — range: other
  • Worst-case geometric violation over the trajectory: $d_{\max} = \max_{t}d_{t}$.
• Conditional Avg Penetration ($\bar{d}_{\text{cond}}$) — range: other
  • Average penetration severity conditioned on collision events: $\bar{d}{\text{cond}} = \frac{\sum{t}d_{t}\mathbf{1}[d_{t}>0]}{\max(1,\sum_{t}\mathbf{1}[d_{t}>0])}$.
• Time-Normalized Penetration ($I_{\text{pd}}$) — range: other
  • Time-normalized penetration magnitude across the full traversal: $I_{\text{pd}} = \frac{1}{T}\sum_{t=1}^{T}d_{t}$.

Input / output format

Input: Joint configurations q_t for each frame t of a trajectory, the scene mesh M, and reference flat-ground kinematic statistics.

Output: Scalar Motion Adaptation Score and four collision safety metrics (R_col, d_max, d_cond, I_pd).

Scoring recipe

def compute_metrics(trajectory_q, scene_mesh, ref_stats):
    # 1. Motion Adaptation Score
    features = extract_subspace_features(trajectory_q) # posture, vertical, foot, smoothness
    adaptation_score = compute_frechet_distance(features, ref_stats)
    
    # 2. Collision Safety Metrics
    penetration_depths = []
    for frame in trajectory_q:
        poses = forward_kinematics(frame)
        samples = generate_surface_samples(poses)
        s = signed_distance_field_query(samples, scene_mesh)
        d_t = max(0, -min(s))
        penetration_depths.append(d_t)
        
    R_col = mean([1 if d > 0 else 0 for d in penetration_depths])
    d_max = max(penetration_depths)
    d_cond = mean([d for d in penetration_depths if d > 0]) if any(d > 0 for d in penetration_depths) else 0
    I_pd = mean(penetration_depths)
    
    return adaptation_score, R_col, d_max, d_cond, I_pd

Common pitfalls

• The adaptation score measures deviation from nominal flat-ground locomotion, not absolute task success or environmental difficulty.
• Collision metrics require signed distance field queries against the original non-convex scene mesh, not simplified collision proxies.
• Subspace distances must be normalized and uniformly weighted before aggregation; raw distances cannot be summed directly.

Evidence (verbatim from paper)

Given a test trajectory, we compute the Fréchet distance between its empirical feature distribution and the baseline distribution within the same subspace. The Fréchet distance captures both mean displacement and covariance shift, enabling joint evaluation of first- and second-order kinematic deviations. Subspace distances are normalized and aggregated via uniform weighting to produce a scalar adaptation score.

Citation

@misc{wang2026mtc,
  title={Moving Through Clutter: Scaling Data Collection and Benchmarking for 3D Scene-Aware Humanoid Locomotion via Virtual Reality},
  author={Wang et al. (2026)},
  year={2026},
  note={arXiv:2603.05993}
}

• arXiv: 2603.05993