Breakout Determinism Eval

breakout-determinism-eval

Deterministic Implementations for Reproducibility in Deep Reinforcement Learning — Nagarajan et al. (2018) (arXiv:1809.05676, 2018)

What this evaluates

Measures the sensitivity of deep Q-learning performance to various sources of nondeterminism (GPU operations, environment stochasticity, exploration seeds, weight initialization, minibatch sampling) by comparing performance variance across controlled experimental groups.

Datasets

• Atari BREAKOUT — total ?; splits: evaluation (100)

Metrics

• mean score (primary) — range: other
  • Average reward accumulated over 100 evaluation episodes.
• standard deviation — range: other
  • Standard deviation of the 100 episode scores.
• relative standard deviation — range: percent
  • (standard deviation / mean score) * 100%.

Input / output format

Input: Grayscale frames from the Atari BREAKOUT environment.

Output: Discrete action from the environment's action space (greedy policy during evaluation).

Scoring recipe

scores = []
for i in range(100):
    if group == 'environment':
        env.reset(sticky_action_seed=unique_seeds[i])
    else:
        env.reset(start_state=unique_start_states[i])
    score = 0
    for step in range(max_steps):
        action = argmax(Q(s))
        s, r, done, _ = env.step(action)
        score += r
        if done: break
    scores.append(score)
mean_score = sum(scores) / len(scores)
std_score = std(scores)
rel_std = (std_score / mean_score) * 100

Common pitfalls

• Using a stochastic policy during evaluation confounds results by mixing policy variance with network variance.
• Typical ALE evaluation uses random start states, but this protocol uses predetermined action sequences to guarantee diverse, non-poor start states.
• Reporting only final network performance ignores the best-performing checkpoint, which is common practice in DRL and masks nondeterminism impact.

Evidence (verbatim from paper)

When we measure the performance of our agents, we want to ensure that any differences in performance are a result of differences between their Q-networks. In doing so, we ensure that we are measuring performance differences due to an individual source of nondeterminism, since the differences between agents' trained Q-networks are solely due to a source of nondeterminism influencing the learning process. As such, we evaluate the agents over 100 episodes (each episode is capped at five minutes of play) using a greedy policy, so that any deviations between agents' policies are a consequence of their different Q-networks. ... We also report the relative standard deviation in performance, in order to provide a domain-agnostic measure of variance, since the numerical score is specific to BREAKOUT.

Citation

@misc{nagarajan2018deterministic,
  title={Deterministic Implementations for Reproducibility in Deep Reinforcement Learning},
  author={Nagarajan et al. (2018)},
  year={2018},
  note={arXiv:1809.05676}
}

• arXiv: 1809.05676