Human Motion Eval

human-motion-eval

Human Motion Modeling using DVGANs — Xiao Lin, Mohamed R. Amer (2018) (arXiv:1804.10652, 2018)

What this evaluates

Evaluates the ability of generative models to synthesize realistic and diverse human motion sequences from text descriptions, as well as their capacity to complete short-term motion sequences given a partial initialization.

Datasets

• Human3.6M (H3.6M) — total 210; splits: train (-1), test (-1)
• CMU Mocap — total 1125; splits: train (757), test (368)

Metrics

• Inception Score (primary) — range: scalar
  • KL(p(action|animation) || p(action)) = H(action) - H(action|animation), where p(action|animation) is the softmax of a trained ranker's matching score between an animation and a pool of action descriptions.
• Animation Retrieval Accuracy (R@K) — range: percent
  • Recall@K measures the fraction of test animations where the ground-truth action description appears in the top-K ranked descriptions from a pool of K candidates.
• Motion Prediction Error — range: other
  • Mean Euclidean distance between predicted and ground-truth joint angles (in exponential map representation) over all joints and time steps.

Input / output format

Input: Text description of an action (for generation) or a short sequence of ground-truth joint angles (2 seconds / 25 frames) for motion completion.

Output: A sequence of 3D body joint angles (represented in exponential map format) corresponding to the requested action or completion length.

Scoring recipe

def compute_inception_score(animations, descriptions):
    scores = ranker.predict(animations, descriptions)
    p_anim_action = softmax(scores, axis=1)
    p_action = mean(p_anim_action, axis=0)
    kl = sum(p_anim_action * log(p_anim_action / p_action), axis=1)
    return exp(mean(kl))

def compute_retrieval_recall(animations, descriptions, k=10):
    scores = ranker.predict(animations, descriptions)
    top_k = argsort(scores, k=k, descending=True)
    return mean([gt_idx in top_k for top_k in top_k]) * 100

def compute_motion_error(pred_angles, gt_angles):
    return mean(l2_norm(pred_angles - gt_angles, axis=-1))

Common pitfalls

• The Inception Score relies on a trained retrieval ranker rather than a standard classifier, making it sensitive to the ranker's architecture and training data.
• Motion prediction error is only evaluated for short-term windows (up to 400ms) and does not capture long-term generation quality or mode collapse.
• Retrieval accuracy pool size K varies by dataset (250 for CMU, 15 for H3.6M), affecting baseline chance levels and cross-dataset comparability.

Evidence (verbatim from paper)

Since our interest is in motion generation from scratch, with focus on the diversity and realism of the samples, we adopt the inception score. It measures the quality of unconditioned generation and the accuracy of action retrieval using the generated videos.

Citation

@misc{lin2018hummotion,
  title={Human Motion Modeling using DVGANs},
  author={Xiao Lin, Mohamed R. Amer (2018)},
  year={2018},
  note={arXiv:1804.10652}
}

• arXiv: 1804.10652