Pub Plot Understanding Eval

pub-plot-understanding-eval

PUB: Plot Understanding Benchmark and Dataset for Evaluating Large Language Models on Synthetic Visual Data Interpretation — Pawelec et al. (2024) (arXiv:2409.02617, 2024)

What this evaluates

This benchmark evaluates multimodal large language models' ability to interpret synthetic data visualizations. It probes their capacity to extract quantitative features, identify statistical properties, and answer specific questions about plots like histograms, time series, boxplots, and violin plots without relying on real-world data contamination.

Datasets

• PUB Synthetic Plot Dataset — total ?; splits: test (-1)

Metrics

• overall score (primary) — range: [0, 1]
  • An aggregate metric computed by combining accuracy or correctness scores across multiple sub-tasks per plot type (e.g., distribution detection, threshold estimation, cluster identification, median/IQR extraction). Scores are reported per plot category and overall.

Input / output format

Input: A synthetic plot image (scatter, histogram, time series, boxplot, or violin) paired with a text prompt specifying the interpretation task (e.g., 'identify the biggest cluster', 'estimate the percentage of data below threshold X').

Output: A natural language text response containing the requested numerical values, identified features, or qualitative assessments.

Scoring recipe

def compute_overall_score(predictions, gold):
    correct_count = 0
    total_tasks = 0
    for pred, gt in zip(predictions, gold):
        pred_vals = extract_values(pred.text)
        for task in gt.subtasks:
            total_tasks += 1
            if matches(pred_vals[task], gt[task], tolerance=0.05):
                correct_count += 1
    return correct_count / total_tasks

Common pitfalls

• Models frequently hallucinate values when visual features like axis scales, density, or color schemes are altered or ambiguous.
• Approximation tasks (e.g., series point approximation) can yield negative overall scores due to large deviations from ground truth.
• Performance varies drastically across plot types, so a single aggregate score may mask severe weaknesses in specific visualization categories.

Evidence (verbatim from paper)

The performance of various models were evaluated based on their ability to identify the biggest cluster, detect cluster centers, and estimate cluster areas. Among the models, claude-3-5-sonnet achieved the highest overall score of 0.682, excelling in identifying the largest cluster and determining cluster centers.

Citation

@misc{pawelec2024pub,
  title={PUB: Plot Understanding Benchmark and Dataset for Evaluating Large Language Models on Synthetic Visual Data Interpretation},
  author={Pawelec et al. (2024)},
  year={2024},
  note={arXiv:2409.02617}
}

• arXiv: 2409.02617