Symbolizer Eval

symbolizer-eval

SYMBOLIZER: Symbolic Model-free Task Planning with VLMs — Azirar et al. (2026) (arXiv:2604.17830, 2026)

What this evaluates

This evaluation probes a VLM's ability to ground visual and textual observations into structured symbolic states (objects, predicates, goals) and subsequently use those representations for effective task and motion planning. It measures both the accuracy of the symbolic grounding pipeline and the end-to-end success rate of classical planners operating on the generated PDDL problem files.

Datasets

• ProDG — total ?; splits: test (-1)
• ViPlan — total ?; splits: test (-1)

Metrics

• F1 (primary) — range: [0, 1]
  • Computed per instance by comparing predicted and ground-truth sets element-wise for objects, predicates, or goal literals. Averaged across all instances.
• Planning success rate — range: percent
  • The fraction of problems for which a valid plan reaching the goal is found after feeding the generated PDDL files to a classical planner.

Input / output format

Input: Visual observation (2D sprites, 3D rendered scenes, or real images) paired with a textual goal specification.

Output: Structured symbolic representation consisting of extracted object sets, predicate sets, and goal literals, formatted as well-formed PDDL problem files consumable by classical planners.

Scoring recipe

def compute_f1(pred_set, gold_set):
    tp = len(pred_set & gold_set)
    fp = len(pred_set - gold_set)
    fn = len(gold_set - pred_set)
    prec = tp / (tp + fp) if (tp + fp) > 0 else 0.0
    rec = tp / (tp + fn) if (tp + fn) > 0 else 0.0
    return 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0.0

def compute_success_rate(predictions, gold_plans):
    correct = sum(1 for p, g in zip(predictions, gold_plans) if is_valid_plan(p, g))
    return (correct / len(predictions)) * 100

Common pitfalls

• Multi-stage grounding pipelines suffer from cascading errors where early detection or captioning mistakes propagate and compound in later predicate and goal extraction stages.
• Static predicates (e.g., clear) are difficult to infer from few-shot examples and are often annotated inconsistently across benchmark instances, leading to variable F1 scores.
• Using planner-feedback retries for goal grounding can degrade accuracy because corrective reprompting lacks fresh visual observation and optimizes for planability rather than true grounding.

Evidence (verbatim from paper)

F1 is computed per instance by comparing predicted and ground-truth sets element-wise, treating each object, predicate instance, or goal literal as individual element, and then averaged across all instances.

Citation

@misc{azirar2026symbolizer,
  title={SYMBOLIZER: Symbolic Model-free Task Planning with VLMs},
  author={Azirar et al. (2026)},
  year={2026},
  note={arXiv:2604.17830}
}

• arXiv: 2604.17830