Cora Crl Eval

cora-crl-eval

CORA: Benchmarks, Baselines, and Metrics as a Platform for Continual Reinforcement Learning Agents — Powers et al. (2021) (arXiv:2110.10067, 2021)

What this evaluates

Evaluates continual reinforcement learning agents across sequential task environments, probing their ability to learn new tasks while retaining old ones (plasticity vs stability) and generalizing to unseen contexts.

Datasets

• Procgen — total ?; splits: train (-1), test (-1)
• MiniHack — total ?; splits: train (-1), test (-1)
• CHORES — total ?; splits: train (-1), test (-1)
• Atari — total ?; splits: train (-1), test (-1)

Metrics

• Continual Evaluation ($\mathcal{C}$) (primary) — range: other (expected return)
  • Mean expected return evaluated on unseen testing environments (solid lines) and training environments (dashed lines) across training intervals. Computed over n steps, w tasks, s cycles per environment as specified in Section 4.
• Isolated Forgetting ($\mathcal{F}$) — range: other (normalized score)
  • Change in performance on previously learned tasks after training on a new task. Negative values indicate backward transfer (forgetting), positive indicate forward transfer.
• Zero-Shot Forward Transfer ($\mathcal{T}$) — range: other (normalized score)
  • Performance on a new task before any training on it, compared to a baseline trained from scratch. Positive values indicate forward transfer.

Input / output format

Input: Visual observations and state information from procedurally generated or simulated environments.

Output: Discrete or continuous action commands depending on the specific environment.

Scoring recipe

# Continual Evaluation (C)
C = mean(expected_return(env=test_envs) during_training)

# Forgetting (F)
F = performance_after_new_task(task_old) - performance_before_new_task(task_old)

# Forward Transfer (T)
T = performance_before_training(task_new) - baseline_performance(task_new)

# Note: n, w, s parameters control steps, tasks, and cycles per environment.

Common pitfalls

• Confusing training vs. test evaluation lines in plots (dashed=train, solid=test).
• Assuming low CHORES returns indicate algorithm failure rather than benchmark difficulty.
• Expecting uniform forward transfer across tasks; transfer varies significantly by task similarity.

Evidence (verbatim from paper)

Three metrics—Continual Evaluation, Isolated Forgetting, and Zero-Shot Forward Transfer—enable rigorous comparison of CRL methods... On the Continual Evaluation plots, solid lines represent evaluation on unseen testing environments, while dashed lines show evaluation on the training environments.

Citation

@misc{powers2021cora,
  title={CORA: Benchmarks, Baselines, and Metrics as a Platform for Continual Reinforcement Learning Agents},
  author={Powers et al. (2021)},
  year={2021},
  note={arXiv:2110.10067}
}

• arXiv: 2110.10067