Calvin Long Horizon Eval

calvin-long-horizon-eval

Latent Plans for Task-Agnostic Offline Reinforcement Learning — Erick Rosete-Beas et al. (2022) (arXiv:2209.08959, 2022)

What this evaluates

Evaluates a robot policy's ability to chain multiple sub-goals sequentially using only visual observations and goal images. It probes long-horizon planning, goal-conditioned control, and the capacity to generalize to unseen goal configurations without explicit reward signals.

Datasets

• CALVIN — total ?; splits: train (-1)

Metrics

• Success rate (primary) — range: percent
  • Percentage of successful rollouts out of the total number of evaluated chains or tasks. A rollout is successful if the agent reaches the specified goal image/state within the timestep limit before failure.
• Average Length — range: other
  • Average number of sub-goals successfully completed in a row before the agent fails or hits the maximum timestep limit.

Input / output format

Input: RGB images from a static camera (simulation) or static + gripper camera (real-world), concatenated with goal images specifying the target state or sub-goal.

Output: 7-DoF robot arm actions (joint velocities/positions).

Scoring recipe

def compute_success_rate(successes, total_rollouts):
    return (successes / total_rollouts) * 100

def evaluate_chain(policy, env, goal_images, max_timesteps=180):
    completed = 0
    for goal in goal_images:
        success = False
        for t in range(max_timesteps):
            action = policy(env.observation, goal)
            env.step(action)
            if env.is_goal_reached(goal):
                success = True
                break
        if success:
            completed += 1
        else:
            break
    return completed, len(goal_images)

Common pitfalls

• Goal images often exclude the robot/end-effector, forcing the model to reason about full scene configuration rather than just end-effector pose.
• Success rates for sequential tasks drop exponentially with chain length; reporting only single-task success masks long-horizon planning capability.
• The 180-timestep limit per sub-goal truncates long tasks, so success rates are highly sensitive to this hyperparameter.

Evidence (verbatim from paper)

We call this evaluation of performing multiple tasks on a row, long-horizon multitask with visual observations LH-MTVis. This setting is very challenging as it requires agents to be able to transition between different subgoals. ... We record the success rate of all models in Table 2. TACO-RL successfully performs long-horizon tasks that require reasoning over sequential behaviors with an final success rate of 27% which corresponds to an order of magnitude improvement upon the LMP and CQL+HER baselines.

Citation

@misc{rosetebeas2022latentplans,
  title={Latent Plans for Task-Agnostic Offline Reinforcement Learning},
  author={Erick Rosete-Beas et al. (2022)},
  year={2022},
  note={arXiv:2209.08959}
}

• arXiv: 2209.08959