Mujoco Continuous Control Eval

mujoco-continuous-control-eval

Benchmarking Deep Reinforcement Learning for Continuous Control — Duan et al. (2016) (arXiv:1604.06778, 2016)

What this evaluates

Evaluates deep reinforcement learning algorithms on continuous control tasks. It probes capabilities in handling high-dimensional state/action spaces, partial observability, sensor noise, delayed actions, and hierarchical decision-making across physics-based simulations.

Datasets

• DeepMind Control Suite (MuJoCo Tasks) — total 31; splits: test (-1); repo https://github.com/rllab/rllab

Metrics

• reward (primary) — range: other
  • Mean cumulative reward over a fixed number of evaluation episodes, averaged across multiple random seeds.

Input / output format

Input: Continuous state vectors (e.g., positions, velocities, joint angles) and optional range sensor readings.

Output: Continuous action vectors applied to the physics simulator at each timestep.

Scoring recipe

def evaluate(policy, env, num_episodes, num_seeds):
    returns = []
    for seed in range(num_seeds):
        env.seed(seed)
        for _ in range(num_episodes):
            obs = env.reset()
            ep_return = 0
            done = False
            while not done:
                action = policy(obs)
                obs, reward, done, _ = env.step(action)
                ep_return += reward
            returns.append(ep_return)
    return np.mean(returns)

Common pitfalls

• RL benchmarks typically do not use fixed train/validation/test splits; performance is reported as average return over random seeds and episodes.
• Tasks include specific penalty terms (e.g., for excessive control or falling) that heavily influence early learning and final performance, requiring careful comparison baselines.

Evidence (verbatim from paper)

The tasks in the presented benchmark can be divided into four categories: basic tasks, locomotion tasks, partially observable tasks, and hierarchical tasks. We briefly describe them in this section. More detailed specifications are given in the supplementary materials and in the source code. The goal for all the tasks is to move forward as quickly as possible. These tasks are more challenging than the basic tasks due to high degrees of freedom. In addition, a great amount of exploration is needed to learn to move forward without getting stuck at local optima. Since we penalize for excessive controls as well as falling over, during the initial stage of learning, when the robot is not yet able to move forward for a sufficient distance without falling, apparent local optima exist including staying at the origin or diving forward slowly.

Citation

@misc{duan2016benchmarking,
  title={Benchmarking Deep Reinforcement Learning for Continuous Control},
  author={Duan et al. (2016)},
  year={2016},
  note={arXiv:1604.06778}
}

• arXiv: 1604.06778