Text To Motion Generation Eval

text-to-motion-generation-eval

Marrying Text-to-Motion Generation with Skeleton-Based Action Recognition — Kuang et al. (2026) (arXiv:2604.17090, 2026)

What this evaluates

Evaluates a unified framework's ability to generate realistic and semantically aligned 3D human motions from text descriptions, recognize actions from skeleton data, and retrieve matching text-motion pairs. It probes the model's semantic fidelity, distributional realism, and cross-modal alignment capabilities.

Datasets

• HumanML3D — total ?; splits: train (-1), test (-1)
• KIT — total ?; splits: train (-1), test (-1)
• NTU-60 — total ?; splits: train (-1), test (-1)
• NTU-120 — total ?; splits: train (-1), test (-1)

Metrics

• R-Precision (primary) — range: [0, 1]
  • Top-K retrieval accuracy where K equals the number of ground-truth text captions per motion (typically 3). Measures semantic alignment between generated motion and text.
• FID — range: other
  • Fréchet Inception Distance computed between the feature distributions of real and generated motion sequences. Lower values indicate higher distributional realism.
• CLIP-score — range: [0, 1]
  • Cosine similarity between text and motion embeddings extracted from a pre-trained CLIP model. Measures fine-grained cross-modal alignment.
• Accuracy — range: percent
  • Top-1 classification accuracy for skeleton-based action recognition, reported across overall, many-shot, medium-shot, and few-shot splits.
• Recall@K — range: percent
  • Percentage of times the correct text-motion or motion-text pair appears in the top-K retrieved results (K=1,2,3,5,10).

Input / output format

Input: Text descriptions of human actions; for generation tasks, the model receives only the text prompt and autoregressively generates absolute joint coordinates.

Output: A sequence of 3D skeleton joint coordinates (absolute positions) representing the generated motion.

Scoring recipe

def compute_r_precision(motions, texts, k=3):
    m_emb = encode_motion(motions)
    t_emb = encode_text(texts)
    sim = m_emb @ t_emb.T
    top_k = sim.argsort(axis=1)[:, -k:]
    gt_indices = np.arange(len(motions))
    hits = np.any(top_k == gt_indices[:, None], axis=1)
    return hits.mean()

def compute_fid(real_motions, gen_motions):
    real_mu, real_sigma = compute_stats(real_motions)
    gen_mu, gen_sigma = compute_stats(gen_motions)
    return fid_score(real_mu, real_sigma, gen_mu, gen_sigma)

Common pitfalls

• Baselines must be re-evaluated on the standardized absolute coordinate system to ensure fair comparison.
• Action recognition baselines must be re-implemented to use the identical visual backbone as the proposed model.
• Cross-dataset generalization requires freezing the MAR backbone and training only a linear classification head (linear probing protocol).

Evidence (verbatim from paper)

To ensure a rigorous and fair comparison, we re-evaluated all baseline methods on the standardized absolute coordinate system. Our unguided model (CoAMD w/o MAR), which leverages the proposed multi-modal representation, already demonstrates highly competitive results outperforming most existing methods. This validates the effectiveness of our foundational architecture. When augmented with our Multi-modal Action Recognizer for guidance (CoAMD (ours)), the model’s performance is substantially boosted, setting new state-of-the-art scores across nearly all metrics on both benchmarks. Most notably, the substantial reduction in FID and the corresponding improvement in CLIP score highlight the dual benefits of our active guidance mechanism, namely enhanced distributional realism of the generated motions and improved fine-grained alignment between motion dynamics and textual semantics.

Citation

@misc{kuang2026marrying,
  title={Marrying Text-to-Motion Generation with Skeleton-Based Action Recognition},
  author={Kuang et al. (2026)},
  year={2026},
  note={arXiv:2604.17090}
}

• arXiv: 2604.17090