Sgplan Eval

sgplan-eval

Contextual Graph Representations for Task-Driven 3D Perception and Planning — Agia (2026) (arXiv:2603.26685, 2026)

What this evaluates

Evaluates classical and neural-symbolic planners on 3D scene graph environments by measuring their ability to generate valid action sequences for task-driven goals within a strict time limit.

Datasets

• SGPlan — total ?; splits: Train (-1), Validation (-1), Test Seen (-1), Test Unseen (-1)

Metrics

• task completion (primary) — range: [0, 1]
  • Fraction of tasks solved within a 10-second timeout. Computed as the number of successful plans divided by the total number of tasks in a given split or category.
• plan time — range: seconds
  • Wall-clock time in seconds required to compute a valid plan. Only averaged over successfully solved tasks, excluding timeouts and failures.
• plan length — range: steps
  • Number of discrete actions/steps in the generated trajectory. Only averaged over successfully solved tasks.

Input / output format

Input: PDDL problem instances (PDDLv2.2 format) representing 3D scene graph environments with robot agents, objects, receptacles, and task goals.

Output: A sequence of actions constituting a valid plan, or a failure/timeout signal if no solution is found within the 10-second limit.

Scoring recipe

def evaluate_planner(planner, problem, timeout=10.0):
    start = time.time()
    plan = planner.solve(problem, timeout=timeout)
    elapsed = time.time() - start
    if plan is not None and elapsed <= timeout:
        return {'task_completion': 1.0, 'plan_time': elapsed, 'plan_length': len(plan)}
    return {'task_completion': 0.0, 'plan_time': elapsed, 'plan_length': None}
# Aggregate by averaging metrics over successful plans per split/category.

Common pitfalls

• Plan time and plan length are only computed and averaged for successfully solved tasks, ignoring timeouts and failures in the mean calculation.
• The strict 10-second timeout disproportionately penalizes planners on long-horizon tasks, masking their actual search efficiency and causing high failure rates.
• Modifications made to the PDDL domain for FF-X compatibility may introduce dead-ends or ill-conditioned states, affecting cross-planner comparability.

Evidence (verbatim from paper)

For each planner, we track task completion as the primary evaluation metric, with plan time and plan length as secondary metrics. In order to fully understand the strengths and limitations of the benchmark, we track these metrics at various levels. By averaging the results over the data splits (i.e., Train, Validation, Test Seen, Test Unseen), we can empirically determine which of the splits contain more challenging problems on average.

Citation

@misc{agia2026contextual,
  title={Contextual Graph Representations for Task-Driven 3D Perception and Planning},
  author={Agia (2026)},
  year={2026},
  note={arXiv:2603.26685}
}

• arXiv: 2603.26685