Partinstruct Eval

partinstruct-eval

PartInstruct: Part-level Instruction Following for Fine-grained Robot Manipulation — Yin et al. (2025) (arXiv:2505.21652, 2025)

What this evaluates

This benchmark probes a robot policy's ability to follow fine-grained, part-level natural language instructions for long-horizon manipulation. It specifically tests zero-shot task decomposition, 3D part grounding, and multi-step planning under varying object, part, and task generalization conditions.

Datasets

• PartInstruct — total ?; splits: train (10000), test (1800)

Metrics

• success (primary) — range: [0, 1]
  • Binary metric per episode. Returns 1 if the agent successfully completes every skill in the predefined chain, and 0 otherwise. The final score is the average success across all evaluation episodes.

Input / output format

Input: Multimodal observations (RGB images, depth maps, point clouds, object/part annotations, proprioceptive joint states and end-effector poses) paired with a single natural language task instruction.

Output: A 7-dimensional action vector per timestep: 3D Cartesian coordinates, roll/pitch/yaw angles, and a single gripper position control.

Scoring recipe

def compute_success(predictions, gold):
    # predictions: sequence of 7D action vectors
    # gold: list of goal predicates for each skill in the chain
    success = True
    for skill_goal in gold:
        if not check_predicate_satisfaction(predictions, skill_goal):
            success = False
            break
    return 1.0 if success else 0.0

Common pitfalls

• Models are trained with explicit skill instructions but evaluated only with high-level task instructions, requiring zero-shot decomposition.
• Success is strictly defined as completing the entire chain of base skills; partial task completion does not count.
• The benchmark tests five distinct generalization conditions (OS, OI, TP, TC, OC); reporting a single aggregate score hides specific failure modes.

Evidence (verbatim from paper)

As defined in Section [III-C], each part-level skill has a binary success criterion. A completion of the entire task means the agent manages to complete every single skill defined in the skill chain.

Citation

@misc{yin2025partinstruct,
  title={PartInstruct: Part-level Instruction Following for Fine-grained Robot Manipulation},
  author={Yin et al. (2025)},
  year={2025},
  note={arXiv:2505.21652}
}

• arXiv: 2505.21652