Ra Dt Icl Eval

ra-dt-icl-eval

Retrieval-Augmented Decision Transformer: External Memory for In-context RL — Schmied et al. (2024) (arXiv:2410.07071, 2024)

What this evaluates

This evaluation probes the in-context learning (ICL) capabilities of reinforcement learning agents across diverse environments, including grid-worlds, robotics simulators, and video games. It measures how effectively an agent can leverage retrieved past experiences to improve its policy over consecutive interaction trials without weight updates.

Datasets

• Dark-Room — total ?; splits: train (80), test (20)
• Dark Key-Door — total ?; splits: train (80), test (20)
• MazeRunner — total ?; splits: train (100), test (20)
• Meta-World — total ?; splits: train (45), test (5)
• DMControl — total ?; splits: train (11), test (5)
• Procgen — total ?; splits: train (12), test (4)

Metrics

• mean reward (primary) — range: other
  • Average cumulative reward obtained over a fixed number of ICL trials (episodes). Reported as the mean across tasks with 95% confidence intervals over 3 random seeds.

Input / output format

Input: State observations (e.g., x-y coordinates, continuous Lidar-like depth vectors, or 64x64 RGB images) concatenated with retrieved sub-trajectories (state, action, reward sequences) or full context windows for baselines.

Output: Discrete action indices (for grid-worlds and Procgen) or continuous action vectors (for Meta-World and DMControl).

Scoring recipe

rewards = []
for trial in range(num_icl_trials):
    episode_reward = 0
    for step in range(max_steps):
        action = model.predict(state, context)
        next_state, reward, done, _ = env.step(action)
        episode_reward += reward
        state = next_state
        if done: break
    rewards.append(episode_reward)
mean_reward = np.mean(rewards)

Common pitfalls

• Confusing standard RL evaluation (single long episode or fixed steps) with ICL evaluation, which requires measuring performance improvement across multiple consecutive trials where past episodes are stored in memory.
• Failing to report 95% confidence intervals over multiple random seeds, as the paper explicitly requires this for fair comparison across methods.
• Mixing up training and evaluation task splits, particularly in Procgen (PG12-Seen vs PG12-Unseen vs PG4) and Meta-World/DMControl, which have distinct holdout sets.

Evidence (verbatim from paper)

We evaluate each agent for 40 episodes on each of the 20 evaluation tasks and report mean reward (+ 95% CI, 3 seeds).

Citation

@misc{schmied2024radt,
  title={Retrieval-Augmented Decision Transformer: External Memory for In-context RL},
  author={Schmied et al. (2024)},
  year={2024},
  note={arXiv:2410.07071}
}

• arXiv: 2410.07071