Planviz Eval

planviz-eval

PlanViz: Evaluating Planning-Oriented Image Generation and Editing for Computer-Use Tasks — Li et al. (2026) (arXiv:2602.06663, 2026)

What this evaluates

Evaluates multimodal models' ability to generate and edit images that correctly follow planning-oriented instructions (route planning, workflow diagramming, web/UI displaying). It probes procedural reasoning, spatial consistency, and semantic alignment in visual synthesis.

Datasets

• PlanViz — total ?; splits: test (-1)

Metrics

• Cor (primary) — range: [0, 1]
  • Score between 0 and 1 measuring whether the generated or edited image correctly follows the planning task instructions and contains useful, accurate content.
• Vis — range: [0, 1]
  • Score between 0 and 1 evaluating the aesthetic and visual coherence of the output image.
• Ef — range: [0, 1]
  • Score between 0 and 1 measuring the efficiency or absence of unwanted parts in the generated image.
• Avg — range: [0, 1]
  • Arithmetic mean of Cor, Vis, and Ef scores.

Input / output format

Input: Text prompts describing planning tasks (route planning, workflow diagramming, or web/UI displaying) for either image generation or image editing.

Output: Generated or edited images.

Scoring recipe

def score_planviz(predictions, gold):
    scores = []
    for img in random.sample(predictions, 50):
        c, v, e = judge_model.evaluate(img, gold) # Qwen3-VL-235B-A22B-Instruct API
        scores.append({'Cor': c, 'Vis': v, 'Ef': e})
    avg_cor = mean([s['Cor'] for s in scores])
    avg_vis = mean([s['Vis'] for s in scores])
    avg_ef = mean([s['Ef'] for s in scores])
    return {'Cor': avg_cor, 'Vis': avg_vis, 'Ef': avg_ef, 'Avg': (avg_cor + avg_vis + avg_ef) / 3}

Common pitfalls

• High visual quality (Vis) scores often mask low correctness (Cor) scores, as models frequently generate visually plausible but semantically misaligned images.
• Thinking/reasoning mechanisms do not consistently improve performance and can sometimes degrade results, especially in editing tasks.
• Editing tasks are significantly harder than generation tasks due to dual constraints (preserving original content while following new instructions).

Evidence (verbatim from paper)

Note that in our experiments, $Cor$ is the most important score to measure whether models can plan tasks and generate with correct and useful images. We randomly select a subset of 50 images from all generations and utilize the Python API of Qwen3-VL-235B-A22B-Instruct, mentioned in Section 3.3 to judge them 10 times. All scores are between 0 and 1.

Citation

@misc{li2026planviz,
  title={PlanViz: Evaluating Planning-Oriented Image Generation and Editing for Computer-Use Tasks},
  author={Li et al. (2026)},
  year={2026},
  note={arXiv:2602.06663}
}

• arXiv: 2602.06663