Semantic Helm Eval

semantic-helm-eval

Semantic HELM: A Human-Readable Memory for Reinforcement Learning — Paischer et al. (2023) (arXiv:2306.09312, 2023)

What this evaluates

Evaluates reinforcement learning agents' ability to learn and utilize memory mechanisms in partially observable environments. It probes sample efficiency, convergence speed, and the capacity to retain and retrieve semantic information across varying levels of visual complexity and task duration.

Datasets

• MiniGrid — total ?; splits: test (-1)
• MiniWorld — total ?; splits: test (-1)
• Avalon — total ?; splits: test (1000)
• Psychlab (CR task) — total ?; splits: test (-1)

Metrics

• IQM (primary) — range: other
  • Interquartile mean of the accumulated reward across random seeds. Computed by discarding the bottom and top 25% of seed results and averaging the remainder.
• Mean Human Normalized Score — range: [0, 1]
  • Agent score normalized by the difference between human performance and a random baseline. Values typically range from 0 to 1, where 1 matches human performance.

Input / output format

Input: RGB image observations per timestep. For memory-based variants, past observations are encoded into semantic language tokens via a frozen CLIP vision encoder and fed to a TransformerXL memory module.

Output: Action selected by the policy (discrete or continuous) to execute in the environment.

Scoring recipe

def compute_iqm(rewards_per_seed):
    sorted_r = sorted(rewards_per_seed)
    n = len(sorted_r)
    lower = int(0.25 * n)
    upper = int(0.75 * n)
    return sum(sorted_r[lower:upper]) / (upper - lower)

def compute_hns(agent_score, human_score, random_score):
    return (agent_score - random_score) / (human_score - random_score)

Common pitfalls

• Partial observability does not automatically imply a need for memory; memory-less PPO can solve MiniGrid and MiniWorld tasks.
• CLIP token retrieval in synthetic environments is biased toward abstract tokens (e.g., 'screenshot', 'biome'), requiring custom prompts for accurate mapping.
• Psychlab CR task performance degrades significantly with low observation resolution due to CLIP token conflation.

Evidence (verbatim from paper)

We report results via IQM (Agarwal et al., 2021) and 95% bootstrapped confidence intervals (CIs) unless mentioned otherwise. The final performance of an agent is measured in terms of mean human normalized scores on a curated set of 1000 test worlds.

Citation

@misc{paischer2023semantichelm,
  title={Semantic HELM: A Human-Readable Memory for Reinforcement Learning},
  author={Paischer et al. (2023)},
  year={2023},
  note={arXiv:2306.09312}
}

• arXiv: 2306.09312