Helm Eval

helm-eval

HELM: Harness-Enhanced Long-horizon Memory for Vision-Language-Action Manipulation — Zijian Zeng et al. (arXiv:2604.18791, 2026)

What this evaluates

Evaluates long-horizon vision-language-action (VLA) manipulation capabilities, specifically testing a model's ability to maintain cross-phase context, predict action failures before execution, and recover from perturbations via rollback or replanning.

Datasets

• LIBERO-LONG — total 500; splits: test (500)
• CALVIN ABC→D — total ?; splits: test (-1)
• LIBERO-Recovery — total ?; splits: test (-1)

Metrics

• TSR (primary) — range: percent
  • Task Success Rate: percentage of episodes where the entire task chain is completed successfully.
• SCR — range: percent
  • Subgoal Completion Rate: percentage of individual subgoals completed successfully out of all attempted subgoals.
• RSR — range: percent
  • Recovery Success Rate: percentage of perturbed episodes where the agent successfully recovers and completes the task after a failure.
• Chains — range: other
  • Average number of completed task chains per episode (maximum 5).

Input / output format

Input: Language instructions, current and past visual observations, and retrieved episodic memory context.

Output: Sequential robot actions (e.g., joint velocities), binary failure predictions, and recovery/rollback decisions.

Scoring recipe

def compute_metrics(episodes):
    tsr = sum(1 for e in episodes if e.success) / len(episodes) * 100
    scr = sum(e.completed_subgoals for e in episodes) / sum(e.total_subgoals for e in episodes) * 100
    rsr = sum(1 for e in episodes if e.perturbed and e.recovered) / sum(1 for e in episodes if e.perturbed) * 100
    chains = sum(e.completed_chains for e in episodes) / len(episodes)
    return {'TSR': tsr, 'SCR': scr, 'RSR': rsr, 'Chains': chains}

Common pitfalls

• RSR is only computed on perturbed episodes in LIBERO-Recovery, not on the standard LIBERO-LONG test set.
• Context length alone (H=32) does not substitute for explicit memory retrieval; increasing H yields diminishing returns compared to the EMM+SV framework.

Evidence (verbatim from paper)

LIBERO-Recovery (ours): LIBERO-LONG with controlled perturbations (object displacement ±5 cm or gripper state flip) injected at a random subgoal boundary; reports Recovery Success Rate (RSR). Table 2: Main results. TSR = Task Success Rate (%), SCR = Subgoal Completion Rate (%), RSR = Recovery Success Rate (%).

Citation

@misc{zeng2026helm,
  title={HELM: Harness-Enhanced Long-horizon Memory for Vision-Language-Action Manipulation},
  author={Zijian Zeng et al.},
  year={2026},
  note={arXiv:2604.18791}
}

• arXiv: 2604.18791