Encqa Eval

encqa-eval

EncQA: Benchmarking Vision-Language Models on Visual Encodings for Charts — Mukherjee et al. (2025) (arXiv:2508.04650, 2025)

What this evaluates

Evaluates vision-language models on their ability to interpret visual encodings (position, length, area, color, shape) and perform chart-specific analytic tasks (e.g., value retrieval, anomaly detection, correlation estimation). It probes fine-grained visual perception, reasoning under different encoding constraints, and whether model capabilities scale with size or prompting strategies.

Datasets

• EncQA — total 2076; splits: test (2076); repo https://github.com/apple/ml-encqa

Metrics

• accuracy (primary) — range: [0, 1]
  • Relaxed accuracy for multiple-choice questions and exact match for numeric responses, reported as the proportion of correctly answered questions across task-encoding combinations.
• sMAPE — range: [0, 1]
  • Symmetric Mean Absolute Percentage Error for numeric responses: sMAPE = (1/n) * sum(|T_k - P_k| / ((|T_k| + |P_k|)/2)), where T_k and P_k are true and predicted values.

Input / output format

Input: Chart images paired with natural language questions (multiple-choice or open-ended numeric) targeting specific visual encodings and analytic tasks.

Output: Model-generated answers: either a selected option for multiple-choice questions or a numerical value for open-ended tasks.

Scoring recipe

def compute_metrics(predictions, ground_truths, task_type):
    if task_type == 'multiple_choice':
        return sum(1 for p, g in zip(predictions, ground_truths) if p == g) / len(predictions)
    elif task_type == 'numeric':
        acc = sum(1 for p, g in zip(predictions, ground_truths) if p == g) / len(predictions)
        smape_vals = [abs(t - p) / ((abs(t) + abs(p)) / 2) for t, p in zip(ground_truths, predictions)]
        return acc, sum(smape_vals) / len(smape_vals)

Common pitfalls

• Model performance does not reliably scale with parameter count or inference compute for many task-encoding pairs.
• Chain-of-Thought prompting often fails to improve performance and can sometimes degrade it on specific task-encoding combinations.
• Models exhibit strong bias in anomaly detection tasks, performing well only when the outlier category has a smaller count than the majority.

Evidence (verbatim from paper)

Two of our tasks require a precise numerical response, namely \taskRetrieve Values and \taskCompute Derived Values. We previously reported relaxed accuracy scores for these tasks ([Figure 2]). While useful, this metric gives limited insight into the degree to which answers are correct or incorrect. In order to provide a finer grained look at model responses, we computed the symmetric mean absolute percentage error (sMAPE) between the model predictions and ground truth for all numeric responses ([Figure 5]). sMAPE is a continuous measure of the distance of a model’s predictions from the true values. It is computed as follows: | | $sMAPE=\frac{1}{n}\sum_{k=1}^{n}\frac{|T_{k}-P_{k}|}{(|T_{k}|+|P_{k}|)/2}$ | | where $T_{k}$ and $P_{k}$ are the true and predicted values respectively of the $k$th question.

Citation

@misc{mukherjee2025encqa,
  title={EncQA: Benchmarking Vision-Language Models on Visual Encodings for Charts},
  author={Mukherjee et al. (2025)},
  year={2025},
  note={arXiv:2508.04650}
}

• arXiv: 2508.04650