Iplotbench Eval

iplotbench-eval

Beyond Pixels: Introspective and Interactive Grounding for Visualization Agents — Lu et al. (2026) (arXiv:2604.21134, 2026)

What this evaluates

Evaluates a visualization agent's ability to reconstruct interactive charts from static images and answer binary questions about them. It probes the agent's capacity for spec-grounded introspection and view-grounded interaction to resolve visual ambiguities like overlapping geometries.

Datasets

• iPlotBench — total 500; splits: test (500); repo https://github.com/HexSys-lab/iPlotBench

Metrics

• Semantic Structural Similarity (S_Type, S_Data, S_Text, S_Style) — range: [0, 1]
  • Measures fidelity of the recreated chart to the reference across four dimensions: trace typing, data fidelity, text extraction, and style. S_Data uses an exponential penalty parameter λ to score deviations from the ground-truth specification.
• Question-level accuracy (primary) — range: [0, 1]
  • Fraction of correctly answered binary questions. Per-figure accuracy averages accuracy within each figure before averaging across figures to avoid overweighting chart types with more questions.

Input / output format

Input: Static reference image of a Plotly chart (Task 1) or the agent's recreated interactive figure plus a binary question (Task 2).

Output: Task 1: Recreated interactive chart code/spec. Task 2: Binary answer (yes/no) to the question.

Scoring recipe

def score_task1(recreated, reference, lambda=5):
    s_data = compute_fidelity(recreated, reference, penalty=lambda)
    s_type, s_text, s_style = compute_component_scores(recreated, reference)
    return s_type, s_data, s_text, s_style

def score_task2(predictions, golds):
    correct = sum(1 for p, g in zip(predictions, golds) if p == g)
    return correct / len(golds)

def per_figure_accuracy(fig_preds, fig_golds):
    fig_accs = [sum(p==g for p,g in zip(ps, gs))/len(gs) for ps, gs in zip(fig_preds, fig_golds)]
    return sum(fig_accs) / len(fig_accs)

Common pitfalls

• Per-figure accuracy is computed by averaging within-figure accuracies first, then across figures, rather than a simple global accuracy, to prevent chart types with more questions from dominating the score.
• Conditional QA accuracy filters for reconstruction fidelity (S_Data >= 0.9), so low scores may reflect poor chart recreation rather than flawed reasoning.
• Tool usage (e.g., zoom, toggle) is optional; agents may rely solely on introspection or interaction, and adding interaction can sometimes distract from spec-level fixes during reconstruction.

Evidence (verbatim from paper)

As shown in Table 4, equipping agents with introspection (+Intro) drives the most significant gains in semantic reconstruction, achieving the highest scores in trace typing (S_Type), data fidelity (S_Data), and style (S_Style). While interaction (+Inter) slightly outperforms in text extraction (S_Text), likely by exposing occluded labels, it provides limited structural benefit.

Citation

@misc{lu2026beyondpixels,
  title={Beyond Pixels: Introspective and Interactive Grounding for Visualization Agents},
  author={Lu et al. (2026)},
  year={2026},
  note={arXiv:2604.21134}
}

• arXiv: 2604.21134