# Halma Eval

> halma-eval

- Skill: `qhjqhj00/halma-eval` (Agent Skill)
- Install (CLI): `npx skillmds@latest add qhjqhj00/halma-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/halma-eval/raw
- Safety review: pending (external: skill-scanner PASS, skillspector PASS)
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-21
- Page: https://skillmd.com/skills/qhjqhj00/halma-eval

---


# halma-eval

> HALMA: Humanlike Abstraction Learning Meets Affordance in Rapid Problem Solving — Xie et al. (2021) (arXiv:2102.11344, 2021)

## What this evaluates

Evaluates an agent's ability to learn humanlike abstractions and affordances for rapid problem solving in a structured visual game. It probes three levels of generalization: perceptual recognition, conceptual abstraction of semantics, and algorithmic strategy formation under limited training exposure.

## Datasets

- **HALMA** — total ?; splits: train (-1), random_split (-1), ST_generalization (-1), AfT_generalization (-1), AnT_generalization (-1)

## Metrics

- `valid_move_ratio ($
ho_a$)` — range: percent
  - The proportion of executed moves that are legally valid within the game rules.
- `goal_reaching ($
ho_g$)` **(primary)** — range: percent
  - The percentage of test episodes in which the agent successfully reaches the designated goal state.
- `efficiency ($
ho_p$)` — range: percent
  - A measure of the agent's path or action efficiency relative to an optimal or maximum allowed length.

## Input / output format

**Input**: Symbolic one-hot observations representing ground-truth game state, or raw visual observation panels.

**Output**: Discrete actions/moves within the Super Halma environment.

## Scoring recipe

```python
def score(predictions, gold):
    valid_ratio = sum(1 for a in predictions if is_valid(a)) / len(predictions)
    goal_reached = sum(1 for ep in predictions if ep.reaches_goal()) / len(predictions)
    efficiency = compute_efficiency(predictions) # e.g., optimal_steps / actual_steps
    return valid_ratio, goal_reached, efficiency
```

## Common pitfalls

- Assuming a high goal-reaching rate implies the agent understands affordances or temporal grammar; low valid move ratios reveal brittle rule comprehension.
- Confusing random split evaluation with true out-of-distribution generalization tests (ST, AfT, AnT) that require compositional abstraction.
- Using standard CNNs without disentanglement priors causes latent space failures that misrepresent the agent's ability to recognize objects and colors.

## Evidence (verbatim from paper)

> All agents show relatively low valid move ratio $m{
ho_{a}}$ in tests of random split, indicating their understanding of affordance is brittle even with the ground-truth semantics. Under this precondition, we find that all agents can still perform relatively well in terms of goal-reaching $m{
ho_{g}}$ and efficiency $m{
ho_{p}}$ in random splits.

## Citation

```bibtex
@misc{xie2021halma,
  title={HALMA: Humanlike Abstraction Learning Meets Affordance in Rapid Problem Solving},
  author={Xie et al. (2021)},
  year={2021},
  note={arXiv:2102.11344}
}
```

- arXiv: 2102.11344