Dvqa Eval

dvqa-eval

DVQA: Understanding Data Visualizations via Question Answering — Kafle et al. (2018) (arXiv:1801.08163, 2018)

What this evaluates

This benchmark evaluates a model's ability to perform visual reasoning and information extraction on bar chart data visualizations. It specifically probes whether systems can accurately read chart-specific labels, handle out-of-vocabulary terms, and answer natural language questions about quantitative relationships and chart structure.

Datasets

• DVQA — total ?; splits: train (-1), Test-Familiar (-1), Test-Novel (-1)

Metrics

• exact-match accuracy (primary) — range: [0, 1]
  • A prediction is counted as correct only if the generated string is character-for-character identical to the ground truth answer.
• edit distance — range: [0, 1]
  • A prediction is counted as correct if the Levenshtein edit distance between the generated string and the ground truth is less than or equal to 1.

Input / output format

Input: A bar chart image and a natural language question.

Output: A single string representing the answer.

Scoring recipe

def exact_match_accuracy(predictions, golds):
    return sum(1 for p, g in zip(predictions, golds) if p == g) / len(golds)

def edit_distance_accuracy(predictions, golds, max_dist=1):
    return sum(1 for p, g in zip(predictions, golds) if edit_distance(p, g) <= max_dist) / len(golds)

Common pitfalls

• Fixed-vocabulary models fail on chart-specific labels because they cannot encode or generate out-of-vocabulary terms present in the visualization.
• Minor OCR or string generation errors cause complete failure under exact-match scoring, especially on the Test-Novel split where labels are unseen during training.

Evidence (verbatim from paper)

To measure performance, an algorithm gets a question correct only if it generates a string that is identical to the ground truth. To better assess MOM, we also measure its performance using edit distance, which is denoted MOM ($\pm 1$). This model is allowed to get a question correct as long as the answer it generates is within one edit distance or less compared to the correct answer.

Citation

@misc{kafle2018dvqa,
  title={DVQA: Understanding Data Visualizations via Question Answering},
  author={Kafle et al. (2018)},
  year={2018},
  note={arXiv:1801.08163}
}

• arXiv: 1801.08163