Karate Attribute Manipulation Eval

karate-attribute-manipulation-eval

Motion Diffusion Autoencoders: Enabling Attribute Manipulation in Human Motion Demonstrated on Karate Techniques — Richardson et al. (2025) (arXiv:2501.18729, 2025)

What this evaluates

Evaluates a model's ability to manipulate specific semantic attributes (e.g., technique, skill level) in human motion data while preserving untargeted attributes and anatomical accuracy. It probes latent space disentanglement and the capacity for controlled, attribute-level motion editing.

Datasets

• Kyokushin karate dataset — total ?; splits: test (-1)

Metrics

• linear separability (primary) — range: other
  • Measures how well different attribute classes are separated in the latent embedding space after projection into two dimensions, following the approach of Karras et al. (2019) for quantifying latent space disentanglement.
• qualitative evaluation — range: other
  • Expert-based assessment comparing manipulated motion samples against expected outcomes derived from domain knowledge to verify anatomical accuracy and attribute preservation.

Input / output format

Input: Preprocessed motion capture sequences of human karate techniques, represented using a rotation-based pose format and encoded into a semantic embedding space.

Output: Manipulated motion sequences with targeted attributes altered, alongside their corresponding 2D-projected latent embeddings for separability analysis.

Scoring recipe

# Linear separability scoring
embeddings_2d = project_to_2d(latent_embeddings)
accuracy = train_linear_classifier(embeddings_2d, attribute_labels).score()

# Qualitative scoring
expert_score = assess_anatomical_accuracy_and_attribute_change(
    manipulated_motions, expected_outcomes
)

# Final metric
metric_value = accuracy  # primary metric

Common pitfalls

• FID and other distribution-based metrics are explicitly invalid here because they require ≥10k samples per attribute group, which the available datasets cannot provide.
• Linear separability is only measured after 2D projection, which may oversimplify high-dimensional disentanglement and misrepresent true attribute independence.
• Qualitative evaluation relies heavily on expert karate knowledge, making it subjective and difficult to reproduce without domain-specific training.

Evidence (verbatim from paper)

As an alternative to distribution-based approaches, Karras et al. (Karras et al., [2019])* propose to quantify the latent space disentanglement by measuring its linear separability. We adapt this idea into the domain of human motion manipulation and further inspect the latent space after projection into two dimensions. Given these limitations at quantifying the models performance on human motion manipulation, we additionally perform a qualitative evaluation, where we first formulate expected outcomes based on expert knowledge and then inspect whether the test manipulations match those expectations.

Citation

@misc{richardson2025modiffae,
  title={Motion Diffusion Autoencoders: Enabling Attribute Manipulation in Human Motion Demonstrated on Karate Techniques},
  author={Richardson et al. (2025)},
  year={2025},
  note={arXiv:2501.18729}
}

• arXiv: 2501.18729