Motion Turing Test Eval

motion-turing-test-eval

Towards Motion Turing Test: Evaluating Human-Likeness in Humanoid Robots — Li et al. (2026) (arXiv:2603.06181, 2026)

What this evaluates

Evaluates a model's ability to predict human-likeness scores for humanoid and human motion sequences based purely on kinematic data. It probes whether models can align with human perceptual judgments of motion fluency, coordination, and naturalness without relying on visual appearance cues.

Datasets

• HHMotion — total 1000; splits: test (-1)

Metrics

• MAE — range: [0, 5]
  • Mean absolute error between predicted and human-annotated scores. Computed as the average of absolute differences across all test instances.
• RMSE — range: [0, 5]
  • Root mean square error between predicted and human-annotated scores. Computed as the square root of the average of squared differences across all test instances.
• Spearman's ρ (primary) — range: [-1, 1]
  • Spearman rank correlation coefficient between predicted and human-annotated scores. Measures the monotonic relationship between model predictions and human ratings.

Input / output format

Input: Rendered SMPL-X motion videos (or sequences of SMPL-X poses) representing human or humanoid actions.

Output: A single float score on a 0–5 scale indicating human-likeness, optionally accompanied by a qualitative description.

Scoring recipe

import numpy as np
from scipy import stats

def compute_metrics(preds, golds):
    preds, golds = np.array(preds), np.array(golds)
    mae = np.mean(np.abs(preds - golds))
    rmse = np.sqrt(np.mean((preds - golds) ** 2))
    rho, _ = stats.spearmanr(preds, golds)
    return {'MAE': mae, 'RMSE': rmse, "Spearman's ρ": rho}

Common pitfalls

• VLMs may produce constant outputs across different prompting strategies, resulting in an undefined or invalid Spearman's ρ (marked as '–' in the paper).
• The benchmark explicitly uses SMPL-X retargeted clips to strip visual appearance bias, so models must rely solely on kinematic and temporal cues rather than rendering style or background context.
• Human-likeness scores are averaged across 30 annotators; comparing model predictions to individual annotator ratings instead of the mean will artificially inflate error metrics.

Evidence (verbatim from paper)

The model receives rendered SMPL-X motion videos as input and rates them on a 0–5 human-likeness scale. ... Overall, PTR-Net consistently outperforms all baselines across metrics, achieving lower MAE and RMSE and higher $ ho$.

Citation

@misc{li2026motionturingtest,
  title={Towards Motion Turing Test: Evaluating Human-Likeness in Humanoid Robots},
  author={Li et al. (2026)},
  year={2026},
  note={arXiv:2603.06181}
}

• arXiv: 2603.06181