Realcqa Eval

realcqa-eval

RealCQA: Scientific Chart Question Answering as a Test-bed for First-Order Logic — Ahmed et al. (2023) (arXiv:2308.01979, 2023)

What this evaluates

Evaluates scientific chart question answering capabilities, specifically testing a model's ability to extract, reason over, and answer questions about real-world scientific charts. It probes first-order logic reasoning and neuro-symbolic capabilities by requiring formal verification of logical inferences from complex visual data.

Datasets

• RealCQA — total 683; splits: test (683); repo https://github.com/cse-ai-lab/RealCQA

Metrics

• accuracy (primary) — range: [0, 1]
  • Exact match accuracy calculated as the number of correctly predicted answers divided by the total number of questions. Reported overall and segmented by answer type (String, Numeric, Rank, Unranked, Binary) and question type (Root, Structure, Retrieval, Reasoning).

Input / output format

Input: Chart image (or extracted visual features) concatenated with a natural language question. Some baselines also receive a pre-extracted gold standard table or OCR text as additional context.

Output: Natural language answer string, which can be a single value, a ranked list, an unranked list, or a binary yes/no response.

Scoring recipe

def compute_accuracy(predictions, gold_answers):
    correct = 0
    for pred, gold in zip(predictions, gold_answers):
        if normalize(pred) == normalize(gold):
            correct += 1
    return correct / len(gold_answers)

Common pitfalls

• Models trained on synthetic charts (e.g., PlotQA, FigureQA) fail to generalize to the out-of-distribution real-world chart distributions in RealCQA.
• Evaluation assigns a score of zero to questions if the required data-table context is missing, which can artificially deflate performance for baselines that do not extract tables.
• Sampling strategies heavily impact test set representativeness; overly aggressive sampling may remove challenging QA pairs, inflating accuracy on list-type questions.

Evidence (verbatim from paper)

We present the performance of VLT5 on the RealCQA test-set, which comprises approximately 683 charts with gold-data table annotation. The evaluation is conducted on the charts in this test-set, with a score of zero assigned to the remaining charts. Specifically, the performance on string type answers improves from 0.008% to 30.68%, and the overall accuracy of QA pairs improves from 23.99% to 31.06%.

Citation

@misc{ahmed2023realcqa,
  title={RealCQA: Scientific Chart Question Answering as a Test-bed for First-Order Logic},
  author={Ahmed et al. (2023)},
  year={2023},
  note={arXiv:2308.01979}
}

• arXiv: 2308.01979