Planetarium Eval

planetarium-eval

Planetarium: A Rigorous Benchmark for Translating Text to Structured Planning Languages — Zuo et al. (2024) (arXiv:2407.03321, 2024)

What this evaluates

Evaluates an LLM's ability to translate natural language planning task descriptions into valid, semantically equivalent Planning Domain Definition Language (PDDL) code. It specifically probes the model's capacity to accurately capture initial states, goal states, and object relationships while adhering to formal planning semantics.

Datasets

• Planetarium — total 145918; splits: train (129975), test (15943); repo https://github.com/BatsResearch/planetarium

Metrics

• equivalence (primary) — range: [0, 1]
  • The proportion of generated PDDL files that are semantically equivalent to the ground truth. Equivalence is defined via a bijective mapping between proposition symbols that preserves states, actions, initial conditions, and goals, verified through scene graph isomorphism after fully specifying open-world goal assumptions.

Input / output format

Input: Natural language text description specifying the initial state and goal state of a planning task.

Output: A PDDL problem file containing objects, predicates, initial state, and goal state.

Scoring recipe

correct = 0
for text, gt_pddl in dataset:
    gen_pddl = model.generate(text)
    # Algorithm 1: transform to scene graphs, fully specify goals, check isomorphism
    if equivalent(gen_pddl, gt_pddl, isPlaceholder=False):
        correct += 1
return correct / len(dataset)

Common pitfalls

• Syntactic string matching or solvability checks fail because multiple PDDL representations can encode the same planning problem.
• PDDL goals use an open-world assumption, requiring implicit trivial propositions to be fully specified before equivalence can be checked.
• Object identity handling varies by task; some require exact object permutation matching, while others treat goal objects as placeholders.

Evidence (verbatim from paper)

The Planetarium dataset includes 145,918 text-to-PDDL pairs derived from the Blocks World, Gripper, and Floor Tile domains. Our dataset captures 25 unique initial and goal state configurations describing 73 different tasks. We propose a definition of equivalence in terms of classical planning problems. The main idea is to find a bijective function between the sets of proposition symbols L of the two problems such that makes the two problems equal. Our algorithm for checking equivalence is summarized in Algorithm 1.

Citation

@misc{zuo2024planetarium,
  title={Planetarium: A Rigorous Benchmark for Translating Text to Structured Planning Languages},
  author={Zuo et al. (2024)},
  year={2024},
  note={arXiv:2407.03321}
}

• arXiv: 2407.03321