Qcaleval Eval

qcaleval-eval

QCalEval: Benchmarking Vision-Language Models for Quantum Calibration Plot Understanding — Cao et al. (2026) (arXiv:2604.25884, 2026)

What this evaluates

Probes vision-language models' ability to interpret quantum calibration plots across diverse visual formats (1D traces, 2D maps, histograms) and perform structured scientific reasoning. It tests capabilities ranging from visual grounding and outcome classification to parameter extraction and operational calibration diagnosis, both in zero-shot and in-context learning settings.

Datasets

• QCalEval — total 243; splits: test (243); repo https://github.com/nvidia/QCalEval

Metrics

• accuracy (primary) — range: [0, 1]
  • Percentage of correctly answered questions across all six question types (Q1–Q6) per sample, averaged across the benchmark. Correctness is determined by exact match or schema-compliant JSON output for each question type.

Input / output format

Input: A calibration plot image paired with experiment-family-specific textual background. For in-context learning evaluation (Q3, Q5, Q6), additional demonstration examples (plots with expert-assigned labels or analyses) from the same family are provided.

Output: Structured responses per question type: Q1 (JSON description of plot type/axes/features), Q2 (4-way classification: Expected behavior, Suboptimal parameters, Anomalous behavior, Apparatus issue), Q3 (scientific analysis text), Q4 (Reliable, Unreliable, or No fit), Q5 (JSON extraction of physical parameters), Q6 (family-specific status code and optional corrective range/action).

Scoring recipe

def compute_accuracy(predictions, gold):
    correct = 0
    total = 0
    for q in ['Q1', 'Q2', 'Q3', 'Q4', 'Q5', 'Q6']:
        pred = predictions[q]
        gold_val = gold[q]
        if q in ['Q1', 'Q5']:
            if json_structurally_equal(pred, gold_val): correct += 1
        elif q in ['Q2', 'Q4', 'Q6']:
            if pred == gold_val: correct += 1
        elif q == 'Q3':
            if matches_expert_analysis(pred, gold_val): correct += 1
        total += 1
    return correct / total

Common pitfalls

• Confusing Q2 (coarse 4-way outcome) with Q6 (family-specific actionable diagnosis), as they operate at different granularities.
• Assuming in-context learning applies to all question types; it is explicitly restricted to Q3, Q5, and Q6.
• Treating all 243 samples as usable for ICL; only 236 are evaluated because scenario types with a single sample cannot provide support examples without reusing the query.

Evidence (verbatim from paper)

QCalEval evaluates VLM capabilities on quantum calibration plots through six question types, assessed under both zero-shot (no demonstrations) and in-context learning (with demonstrations) settings. revealing that even top general-purpose VLMs achieve only 72.3 mean accuracy in zero-shot settings

Citation

@misc{cao2026qcaleval,
  title={QCalEval: Benchmarking Vision-Language Models for Quantum Calibration Plot Understanding},
  author={Cao et al. (2026)},
  year={2026},
  note={arXiv:2604.25884}
}

• arXiv: 2604.25884