Ppo Rl Benchmark Eval

ppo-rl-benchmark-eval

Proximal Policy Optimization Algorithms — Schulman et al. (2017) (arXiv:1707.06347, 2017)

What this evaluates

Evaluates reinforcement learning algorithms on continuous control and pixel-based Atari tasks to measure sample efficiency, stability, and final performance. It probes the ability of policy optimization methods to learn effective control policies across diverse physics simulators and arcade games.

Datasets

• OpenAI Gym (MuJoCo) — total ?; splits: all (-1)
• Roboschool — total ?; splits: all (-1)
• Arcade Learning Environment — total 49; splits: all (-1)

Metrics

• average total reward of the last 100 episodes (primary) — range: [0, 1]
  • Mean of total rewards collected in the final 100 episodes of training. Shifted and scaled per environment so that a random policy yields 0 and the best known result yields 1.
• average reward per episode over entire training period — range: other
  • Mean of total rewards collected across all episodes during the full training run.

Input / output format

Input: Environment state observations (continuous vectors for MuJoCo/Roboschool, pixel frames for Atari).

Output: Action sampled from a parameterized policy distribution (Gaussian mean/std for continuous control, categorical for Atari).

Scoring recipe

def compute_normalized_score(rewards_last_100, random_baseline, best_known):
    avg_reward = sum(rewards_last_100) / 100
    normalized = (avg_reward - random_baseline) / (best_known - random_baseline)
    return normalized

final_score = mean([
    compute_normalized_score(run_rewards, env_random, env_best)
    for run in all_runs
])

Common pitfalls

• Normalization baselines (random policy score and best known score) vary by environment and implementation, affecting cross-paper comparability.
• RL results are highly stochastic; the paper averages over 21 runs (7 envs × 3 seeds) for MuJoCo, but only 3 trials for Atari, which may underrepresent variance.
• Atari scoring uses average reward per episode, which can be misleading if episode lengths vary drastically due to frame-skip or early termination.

Evidence (verbatim from paper)

We scored each run of the algorithm by computing the average total reward of the last 100 episodes. We shifted and scaled the scores for each environment so that the random policy gave a score of 0 and the best result was set to 1, and averaged over 21 runs to produce a single scalar for each algorithm setting.

Citation

@misc{schulman2017proximal,
  title={Proximal Policy Optimization Algorithms},
  author={Schulman et al. (2017)},
  year={2017},
  note={arXiv:1707.06347}
}

• arXiv: 1707.06347