Procgen Competition Eval

procgen-competition-eval

Measuring Sample Efficiency and Generalization in Reinforcement Learning Benchmarks: NeurIPS 2020 Procgen Benchmark — Mohanty et al. (2021) (arXiv:2103.15332, 2021)

What this evaluates

Evaluates reinforcement learning agents on their ability to learn efficiently and generalize to unseen, procedurally generated environments. It measures how well algorithms adapt to novel level distributions under strict computational and timestep constraints.

Datasets

• Procgen Benchmark — total 16; splits: train (12), test (4); repo https://github.com/openai/procgen

Metrics

• mean normalized return (primary) — range: [0, 1]
  • The arithmetic mean of normalized returns across all evaluated environments. Normalized return is calculated per environment as (agent_return - random_agent_return) / (perfect_agent_return - random_agent_return), averaged over multiple rollouts.

Input / output format

Input: Stacked visual frames (e.g., 4-frame stacks) from the procedurally generated environment, optionally augmented with additional channels like accumulated episode reward or frame differences.

Output: Discrete action per timestep, sampled from the agent's policy network.

Scoring recipe

def compute_mean_normalized_return(envs, agent, num_rollouts=10):
    normalized_returns = []
    for env in envs:
        returns = []
        for _ in range(num_rollouts):
            obs = env.reset()
            total_reward = 0
            done = False
            while not done:
                action = agent.act(obs)
                obs, reward, done, _ = env.step(action)
                total_reward += reward
            returns.append(total_reward)
        avg_ret = np.mean(returns)
        norm_ret = (avg_ret - random_baseline) / (perfect_baseline - random_baseline)
        normalized_returns.append(norm_ret)
    return np.mean(normalized_returns)

Common pitfalls

• Training or evaluating on the hold-out (extra) environments during the training phase, which violates the generalization constraint.
• Failing to normalize returns against the official random and perfect agent baselines provided by the Procgen benchmark.
• Ignoring the strict 8M timestep limit per environment, which artificially inflates sample efficiency scores.

Evidence (verbatim from paper)

We used the mean normalized return to compare submissions based on a single score across multiple Procogen environments. The mean normalized rewards across multiple rollouts per environment for the top ten submission for sample-efficiency and generalization tracks are shown in 3 and 3.

Citation

@misc{mohanty2021procgen,
  title={Measuring Sample Efficiency and Generalization in Reinforcement Learning Benchmarks: NeurIPS 2020 Procgen Benchmark},
  author={Mohanty et al. (2021)},
  year={2021},
  note={arXiv:2103.15332}
}

• arXiv: 2103.15332