Plotchain Eval

plotchain-eval

PlotChain: Deterministic Checkpointed Evaluation of Multimodal LLMs on Engineering Plot Reading — Ravishankara (2026) (arXiv:2602.13232, 2026)

What this evaluates

This benchmark evaluates multimodal LLMs on engineering plot reading and visual quantitative reasoning. It probes the model's ability to interpret complex axes (including log scales), read curve values, and compute derived engineering quantities like cutoff frequencies or settling times from rendered plot images.

Datasets

• PlotChain — total 450; splits: test (450)

Metrics

• field-level accuracy (primary) — range: [0, 1]
  • Percentage of correctly predicted numeric fields across all final and checkpoint targets, where correctness is determined by a tolerance-based scoring rule (value must fall within a family/field-specific tolerance window). Averaged across all fields.

Input / output format

Input: Image of an engineering plot and a natural-language question.

Output: A single JSON object containing numeric values (or null) for specified final fields and checkpoint fields (prefixed cp_).

Scoring recipe

def compute_field_accuracy(pred_json, gold_json):
    correct = 0
    total = 0
    for field, gold_val in gold_json.items():
        pred_val = pred_json.get(field)
        if pred_val is not None and gold_val is not None:
            total += 1
            # Tolerance is family/field-specific per paper protocol
            tol = get_tolerance(field)
            if abs(float(pred_val) - float(gold_val)) <= tol:
                correct += 1
    return correct / total if total > 0 else 0.0

Common pitfalls

• Models often struggle with log-scale axes and non-linear curve interpolation, leading to systematic reading errors.
• Tolerance thresholds are not uniform; they vary by plot family and field precision, so applying a single global epsilon will mis-score results.
• Checkpoint fields (cp_*) isolate intermediate reasoning steps; models may produce correct final answers via lucky guessing but fail diagnostics, masking true capability gaps.

Evidence (verbatim from paper)

Under strict JSON numeric output and tolerance-based scoring, top models achieve 80.42% average field-level accuracy, but frequency-domain tasks (e.g., bandpass response, FFT spectrum) remain weak, revealing brittleness in visual quantitative reasoning.

Citation

@misc{ravishankara2026plotchain,
  title={PlotChain: Deterministic Checkpointed Evaluation of Multimodal LLMs on Engineering Plot Reading},
  author={Ravishankara (2026)},
  year={2026},
  note={arXiv:2602.13232}
}

• arXiv: 2602.13232