Visplotbench Eval

visplotbench-eval

VisCoder2: Building Multi-Language Visualization Coding Agents — Ni et al. (2025) (arXiv:2510.23642, 2025)

What this evaluates

Evaluates the ability of coding agents to generate executable visualization code across multiple programming languages and chart families, including iterative self-debugging capabilities. It probes both initial code generation fidelity and the model's capacity to recover from execution errors using feedback logs.

Datasets

• VisPlotBench — total 888; splits: test (888)

Metrics

• Execution Pass Rate (primary) — range: [0, 1]
  • Binary metric: 1 if the submitted code executes without error in an isolated runtime and produces a valid rendered image, 0 otherwise.
• Task Score — range: [0, 1]
  • LLM-judged compliance score based on semantic and structural rubrics derived from the natural language instruction.
• Visual Score — range: [0, 1]
  • LLM-judged perceptual similarity score comparing the generated visualization against the reference output.

Input / output format

Input: Natural language instruction following a five-part schema (Setup, Plot Instruct, Data Instruct, Task Description, Style Description) paired with minimal data previews. For self-debug rounds, the input also includes the model's prior code and an excerpt of the execution log.

Output: Executable source code in the target programming language.

Scoring recipe

def evaluate(predictions, gold):
    # Execute in isolated runtime with language-specific kernel/renderer
    exec_log, rendered_image = execute(predictions)
    
    # Execution Pass Rate
    pass_rate = 1.0 if (exec_log.success and rendered_image.valid) else 0.0
    
    # LLM Judge Scores
    task_score = llm_judge(predictions, gold.instruction, rubric='semantic_structural')
    visual_score = llm_judge(rendered_image, gold.reference_image, rubric='perceptual_similarity')
    
    # Multi-round protocol: up to 3 rounds, report best attempt
    best_pass = max(p['pass_rate'] for p in predictions)
    best_task = max(p['task_score'] for p in predictions)
    best_visual = max(p['visual_score'] for p in predictions)
    
    return best_pass, best_task, best_visual

Common pitfalls

• Execution must use language-specific kernels or headless renderers in isolated environments; standard interpreters may fail or produce different outputs.
• LLM judges require strict semantic and structural rubrics to avoid hallucinated compliance scores or bias toward specific coding styles.
• Final evaluation uses the best attempt across up to three self-debug rounds, not the last generated code, which can mislead if only the final output is scored.

Evidence (verbatim from paper)

Execution Pass Rate checks whether the code runs without error and produces a valid visualization. Task Score measures instruction compliance using an LLM judge guided by semantic and structural rubrics, and Visual Score assesses perceptual similarity between generated and reference outputs. Both follow the GPT-based judging protocol of PandasPlotBench.

Citation

@misc{ni2025viscoder2,
  title={VisCoder2: Building Multi-Language Visualization Coding Agents},
  author={Ni et al. (2025)},
  year={2025},
  note={arXiv:2510.23642}
}

• arXiv: 2510.23642