Smplolympics Eval

smplolympics-eval

SMPLOlympics: Sports Environments for Physically Simulated Humanoids — Luo et al. (2024) (arXiv:2407.00187, 2024)

What this evaluates

Evaluates the ability of physically simulated humanoid agents to perform complex, long-horizon Olympic sports tasks using different control policies and motion priors. It probes task completion accuracy, physical realism, and the effectiveness of adversarial vs. hierarchical reinforcement learning in sparse-reward simulation environments.

Datasets

• SMPLOlympics Sports Environments — total ?; splits: test (-1)

Metrics

• Suc Rate (primary) — range: percent
  • Binary metric indicating whether an agent successfully completes a sport task according to its official rules. Averaged over 1000 evaluation trials.
• Avg Dis — range: other
  • Mean distance traveled by the agent or object (e.g., ball, javelin) across evaluation trials, measured in meters.
• Error Dis — range: other
  • Mean Euclidean distance between the intended target location and the actual landing spot of the agent or object.
• Avg Hits — range: other
  • Average number of successful ball strikes recorded per trial for racket sports like tennis and table tennis.
• Hit Rate — range: percent
  • Percentage of trials where the agent successfully strikes the ball with the club in golf tasks.

Input / output format

Input: High-dimensional state observations from Isaac Gym simulation including joint positions, velocities, and environmental cues (e.g., ball/object positions, target coordinates) for SMPL (69D action) or SMPL-X (153D action) humanoids.

Output: Action vector (joint torques/commands) sampled at 30 Hz, dimension 69 for SMPL or 153 for SMPL-X.

Scoring recipe

def evaluate_trials(predictions, gold):
    suc_rates = [1.0 if p['completed_task'] else 0.0 for p in predictions]
    avg_dis = sum(p['distance'] for p in predictions) / len(predictions)
    error_dis = sum(p['target_error'] for p in predictions) / len(predictions)
    avg_hits = sum(p['strikes'] for p in predictions) / len(predictions)
    hit_rate = sum(1 for p in predictions if p['club_hit']) / len(predictions)
    return {
        'Suc Rate': sum(suc_rates) * 100,
        'Avg Dis': avg_dis,
        'Error Dis': error_dis,
        'Avg Hits': avg_hits,
        'Hit Rate': hit_rate * 100
    }

Common pitfalls

• Reward conflict: Policies like AMP may prioritize adversarial style rewards over task completion, leading to unnatural motions or failure to execute the sport (e.g., walking slowly to maximize discriminator reward).
• Curriculum dependency: Tasks with sparse rewards (e.g., high jump, hurdles) will fail to converge without a curriculum learning schedule, as policies get stuck in local minima.
• Pose estimation limits: State-of-the-art video pose estimation fails for fast-moving players/cameras, requiring fallback to AMASS locomotion data for reference motions.

Evidence (verbatim from paper)

Specifically, success rate (Suc Rate) determines whether an agent completes a sport according to set rules. Average distance (Avg Dis) indicates the extent an agent or object travels. For sports involving ball hits, such as tennis and table tennis, we record the average number of successful ball strikes (Avg Hits). Error distance (Error Dis) measures the distance between the intended target and the actual landing spot, applicable in sports like golf, tennis, and penalty kicks. Evaluations are performed on 1000 trials.

Citation

@misc{luo2024smplolympics,
  title={SMPLOlympics: Sports Environments for Physically Simulated Humanoids},
  author={Luo et al. (2024)},
  year={2024},
  note={arXiv:2407.00187}
}

• arXiv: 2407.00187