Rl History Compression Eval

rl-history-compression-eval

History Compression via Language Models in Reinforcement Learning — Paischer et al. (2022) (arXiv:2205.12258, 2022)

What this evaluates

Evaluates the sample efficiency and memory capabilities of reinforcement learning agents in partially observable environments. It probes whether a frozen language model can effectively compress historical observations to enable generalizable task solving without extensive finetuning.

Datasets

• RandomMaze — total ?; splits: (unstated)
• Minigrid (KeyCorridor) — total ?; splits: (unstated)
• Procgen (Memory Mode) — total ?; splits: (unstated)

Metrics

• IQM of return (primary) — range: other
  • Interquartile Mean of the episode return, computed over the last 100 episodes of training. It is calculated by taking the mean of the middle 50% of returns to reduce outlier sensitivity.
• Normalized return — range: other
  • Return normalized between a lower bound (Impala-PPO on fully masked observations) and an upper bound (maximum possible score for the environment).

Input / output format

Input: Flattened grayscale observations (e.g., 9x9 agent-centered crop for RandomMaze, cropped views for Procgen memory mode) or RGB images for baselines. Partially observable views are provided per timestep.

Output: Action selection via an actor-critic policy network (PPO/PPG).

Scoring recipe

def compute_iqm(returns):
    s = sorted(returns)
    n = len(s)
    return np.mean(s[n//4 : 3*n//4])

# Per seed
iqm_per_seed = [compute_iqm(last_100_episode_returns) for _ in range(num_seeds)]
# Aggregate
mean_iqm = np.mean(iqm_per_seed)
ci_95 = bootstrap_confidence_interval(iqm_per_seed, n=1000)
# Significance vs baseline
p_val = wilcoxon_rank_sum_test(iqm_per_seed, baseline_iqms, alternative='greater')

Common pitfalls

• Procgen's 'memory mode' alters level generation and crops observations, making it strictly partially observable; results are not directly comparable to standard distribution modes.
• Markovian baselines (CNN-PPO) can sometimes outperform memory-based methods in reactive tasks (e.g., dodgeball) due to spurious correlations, misleading sample efficiency claims.
• Normalized returns in Procgen can be negative if baselines perform worse than fully masked observations, requiring careful interpretation of the normalization bounds.

Evidence (verbatim from paper)

We evaluate for sample efficiency by measuring the performance at the end of training and test for statistical significance via a one-sided Wilcoxon rank-sum test at a confidence level of α=0.05. The performance is evaluated by measuring the interquartile mean (IQM) and 95 % bootstrapped confidence intervals (CIs), as proposed in Agarwal et al. (2021).

Citation

@misc{paischer2022historycompression,
  title={History Compression via Language Models in Reinforcement Learning},
  author={Paischer et al. (2022)},
  year={2022},
  note={arXiv:2205.12258}
}

• arXiv: 2205.12258