Lifelong Rl Eval

lifelong-rl-eval

Lifelong Reinforcement Learning with Modulating Masks — Ben-Iwhiwhu et al. (2022) (arXiv:2212.11110, 2022)

What this evaluates

Evaluates the ability of reinforcement learning agents to sequentially learn multiple tasks while retaining prior knowledge, generalizing to unseen environments, and leveraging forward transfer from previous tasks. It probes parameter isolation, knowledge composition, and robustness across discrete and continuous action spaces with varying reward and input distributions.

Datasets

• ProcGen — total ?; splits: train (200), test (-1)
• CT-graph — total ?; splits: train (102400), test (-1)
• Minigrid — total ?; splits: train (-1), test (-1)
• Continual World — total ?; splits: train (-1), test (-1)

Metrics

• Total evaluation return (primary) — range: other
  • Area under the curve (AUC) of the lifelong evaluation plot, computed as the average sum of reward obtained across all tasks in the curriculum over training steps.
• Forward transfer — range: other
  • Normalized difference between the AUC of the training plot for the lifelong learning agent and the AUC for the reference single task expert.
• Success rate — range: [0, 1]
  • Binary metric where 1 is awarded if an agent solves a task and 0 otherwise, averaged over episodes.

Input / output format

Input: High-dimensional RGB observations (ProcGen), 2D patterned images/state grids (CT-graph, Minigrid), or visual screenshots/robot states (Continual World), paired with discrete or continuous action spaces.

Output: Discrete or continuous action selection per environment timestep.

Scoring recipe

def compute_metrics(reward_curves, single_task_aucs):
    # Total evaluation return: AUC of average reward across tasks over time
    total_eval = np.trapz(np.mean(reward_curves, axis=0), x=range(len(reward_curves[0])))
    
    # Forward transfer: normalized difference vs single-task expert
    fwd_transfer = []
    for i, curve in enumerate(reward_curves):
        auc_lifelong = np.trapz(curve, x=range(len(curve)))
        fwd_transfer.append((auc_lifelong - single_task_aucs[i]) / single_task_aucs[i])
        
    # Success rate (Continual World)
    success_rate = np.mean([1.0 if solved else 0.0 for solved in task_solutions])
    return total_eval, np.mean(fwd_transfer), success_rate

Common pitfalls

• MaskRI learns tasks independently, so it intentionally lacks a forward transfer metric and is omitted from those comparisons; including it would be incorrect.
• ProcGen evaluation uses procedurally generated levels unseen during training, so 'test tasks' performance measures generalization rather than memorization of training levels.
• Metrics are averaged over multiple seeds (3 or 5) and tasks, with 95% confidence intervals reported; single-run values should not be used for comparison.

Evidence (verbatim from paper)

The metrics report a lifelong evaluation across all tasks at different points during the lifelong training, computed as the average sum of reward obtained across all tasks in the curriculum. The area under the curve (AUC) is reported in corresponding tables. A forward transfer metric, following the formulation employed in Wołczyk et al. (2021), is computed for the CT-graph, Minigrid and Continual World. For each task, the forward transfer is computed as the normalized difference between the AUC of the training plot for the lifelong learning agent and the AUC for the reference single task expert.

Citation

@misc{ben-iwhiwhu2022lifelong,
  title={Lifelong Reinforcement Learning with Modulating Masks},
  author={Ben-Iwhiwhu et al. (2022)},
  year={2022},
  note={arXiv:2212.11110}
}

• arXiv: 2212.11110