Simbav2 Continuous Control Eval

simbav2-continuous-control-eval

Hyperspherical Normalization for Scalable Deep Reinforcement Learning — Hojoon Lee et al. (arXiv:2502.15280, 2025)

What this evaluates

Evaluates the sample efficiency, optimization stability, and scaling capabilities of deep reinforcement learning algorithms across diverse continuous control environments.

Datasets

• MuJoCo — total 5; splits: test (-1)
• DMC Suite — total 21; splits: test (-1)
• MyoSuite — total 10; splits: test (-1)
• HumanoidBench — total 14; splits: test (-1)

Metrics

• Normalized Return (primary) — range: [0, 1]
  • Domain-specific normalization: MuJoCo returns divided by TD3 baseline, DMC returns divided by 1000, MyoSuite uses success rates, HumanoidBench normalized by success score. Final metric is the average across all 57 tasks.

Input / output format

Input: Continuous state observations from MuJoCo, DMC, MyoSuite, and HumanoidBench environments.

Output: Continuous action vectors selected by the policy network.

Scoring recipe

def normalize_return(env, raw_return):
    if env == 'MuJoCo': return raw_return / td3_baseline
    elif env == 'DMC': return raw_return / 1000.0
    elif env == 'MyoSuite': return success_rate
    elif env == 'HumanoidBench': return raw_return / success_score
    return raw_return

scores = [normalize_return(env, get_final_return(env)) for env in all_57_tasks]
metric_value = mean(scores)
ci = bootstrap_ci(scores, n=1000) # 95% CI over seeds

Common pitfalls

• Normalization baselines differ per domain (TD3 for MuJoCo, 1000 for DMC, success rate for MyoSuite, success score for HBench); using a uniform baseline will skew results.
• The UTD (update-to-data) ratio critically affects performance; high UTD ratios require careful handling of weight reinitialization to avoid overfitting, which varies across algorithms.
• Statistical significance is determined via Welch’s t-test (p=0.05), not standard Student’s t-test.

Evidence (verbatim from paper)

To aggregate performance across diverse domains, each environment’s return is normalized to a near $[0,1)$ range. Specifically, MuJoCo performance is normalized by TD3; DMC returns are divided by 1000; MyoSuite scores use success rates; and HumanoidBench scores are normalized by their success score.

Citation

@misc{lee2025hyperspherical,
  title={Hyperspherical Normalization for Scalable Deep Reinforcement Learning},
  author={Hojoon Lee et al.},
  year={2025},
  note={arXiv:2502.15280}
}

• arXiv: 2502.15280