Isaacsim Kitchen Eval

isaacsim-kitchen-eval

Mind and Motion Aligned: A Joint Evaluation IsaacSim Benchmark for Task Planning and Low-Level Policies in Mobile Manipulation — Kachaev et al. (2025) (arXiv:2508.15663, 2025)

What this evaluates

Evaluates a robot's ability to decompose high-level language instructions into executable task plans and execute them in a simulated kitchen environment. It jointly measures planning accuracy and low-level control success under strict time and spatial constraints.

Datasets

• IsaacSim Kitchen Benchmark — total ?; splits: test (-1)

Metrics

• EM (primary) — range: [0, 1]
  • Exact Match accuracy for task planning steps. Calculated as the average per-plan accuracy where each step must exactly match the ground-truth string character-by-character.
• SR — range: [0, 1]
  • Success Rate for mobile manipulation tasks. Equals 1 if a task completes within 120 seconds and meets spatial criteria (navigation: base within 10 cm of target; manipulation: object within 5 cm of target), else 0.
• MSE — range: other
  • Mean Squared Error between the predicted trajectory and the expert trajectory, averaged across all predicted trajectories.

Input / output format

Input: High-level natural language instruction describing a kitchen task.

Output: For planning: a sequence of text strings representing task steps. For control: a predicted trajectory or action sequence for each task step.

Scoring recipe

def compute_joint_metric(predictions, ground_truth, instructions):
    total_score = 0.0
    N = len(instructions)
    for i, instr in enumerate(instructions):
        pred_plan = predictions['plan'][i]
        gt_plan = ground_truth['plan'][i]
        n_i = len(gt_plan)
        em_i = sum(1 for p, g in zip(pred_plan, gt_plan) if p == g) / n_i
        sr_i = 0.0
        for task in gt_plan:
            if execute_task_with_policy(task, predictions['policy']) and time < 120s:
                if check_spatial_criteria(task):
                    sr_i += 1.0
        sr_i /= n_i
        total_score += (em_i + sr_i)
    return total_score / N

Common pitfalls

• Evaluating the controller using the model's predicted plan instead of the ground-truth plan, which conflates planning failures with execution failures.
• Ignoring the strict 120-second time limit per task, which automatically marks a task as failed (SR=0) regardless of spatial proximity.
• Using fuzzy string matching for plan steps; the benchmark requires exact character-by-character matches.

Evidence (verbatim from paper)

For the online joint validation of the task planning and mobile manipulation subproblems in the simulator, we use the EM and Success Rate (SR) metrics. The SR is equal to 1 for a decomposed task if the task is completed successfully within the 120-second time limit for model inference. Success is defined as follows. For a navigation task: the robot’s base geometric center is within 10 cm of the target position. For a manipulation task: the geometric center of the manipulated object is within 5 cm of its target position.

Citation

@misc{kachaev2025mindandmotion,
  title={Mind and Motion Aligned: A Joint Evaluation IsaacSim Benchmark for Task Planning and Low-Level Policies in Mobile Manipulation},
  author={Kachaev et al. (2025)},
  year={2025},
  note={arXiv:2508.15663}
}

• arXiv: 2508.15663