Gqa Eval

gqa-eval

GQA: A New Dataset for Real-World Visual Reasoning and Compositional Question Answering — Hudson et al. (2019) (arXiv:1902.09506, 2019)

What this evaluates

Evaluates visual reasoning and compositional question answering on real-world images. It probes a model's ability to understand scene relationships, answer multi-step questions, and maintain logical consistency across related queries.

Datasets

• GQA — total 22669678; splits: test (-1)

Metrics

• Accuracy (primary) — range: [0, 1]
  • Standard exact-match accuracy: the fraction of questions where the model's predicted answer exactly matches the gold answer.
• Consistency — range: [0, 1]
  • Measures accuracy over entailed questions derived from correctly answered question-answer pairs. For each correctly answered question, the model is tested on questions whose answers can be unambiguously inferred from the original answer.
• Validity — range: [0, 1]
  • Checks whether the predicted answer falls within the valid scope of the question type (e.g., a color for a color question).
• Plausibility — range: [0, 1]
  • Measures whether the predicted answer is reasonable given the question's subject by checking if the answer occurs at least once with that subject across the dataset.
• Distribution — range: other
  • Chi-Square statistic measuring the match between the true answer distribution and the model's predicted answer distribution. Lower values indicate better performance.
• Grounding — range: [0, 1]
  • For attention-based models, the average visual attention probability over the predefined visual region referenced by the question or answer.

Input / output format

Input: A natural language question paired with a corresponding image.

Output: A single word or short phrase representing the answer.

Scoring recipe

def score_accuracy(preds, golds):
    return sum(1 for p, g in zip(preds, golds) if p == g) / len(golds)

def score_consistency(questions, preds, golds, get_entailed, predict, gold_answers):
    correct = [(q, p, g) for q, p, g in zip(questions, preds, golds) if p == g]
    total_entailed = 0
    correct_entailed = 0
    for q, p, g in correct:
        for eq in get_entailed(q, p):
            total_entailed += 1
            if predict(eq) == gold_answers(eq):
                correct_entailed += 1
    return correct_entailed / total_entailed if total_entailed > 0 else 0

def score_validity(questions, preds):
    return sum(1 for q, p in zip(questions, preds) if is_valid_scope(q, p)) / len(questions)

def score_plausibility(questions, preds, subject_answers):
    return sum(1 for q, p in zip(questions, preds) if p in subject_answers[q.subject]) / len(questions)

def score_distribution(true_dist, pred_dist):
    return chi2_statistic(true_dist, pred_dist)  # lower is better

def score_grounding(questions, golds, model_attention, region_pointer):
    scores = [model_attention(q, img)[region_pointer(q, g)] for q, g in zip(questions, golds)]
    return sum(scores) / len(scores)

Common pitfalls

• Models frequently exploit answer priors rather than visual reasoning, inflating accuracy on biased subsets.
• The Distribution metric uses a Chi-Square statistic where lower values indicate better performance, contrary to most accuracy-based metrics.
• Grounding is only applicable to attention-based models and requires predefined visual region pointers for questions/answers.
• Consistency evaluation requires tracking entailed questions across the dataset, which is computationally intensive and not standard in VQA benchmarks.

Evidence (verbatim from paper)

Apart from the standard accuracy metric and the more detailed type-based diagnosis our dataset supports, we introduce five new metrics to get further insight into visual reasoning methods and point to missing capabilities we believe coherent reasoning models should possess.

Consistency. This metric measures responses consistency across different questions. Recall that in section3.3, we used the questions’ semantic representation to derive equivalence and entailment relations between them. When being presented with a new question, any learner striving to be trustworthy should not contradict its previous answers. It should not answer green to a new question about an apple it has just identified as red.

Citation

@misc{hudson2019gqa,
  title={GQA: A New Dataset for Real-World Visual Reasoning and Compositional Question Answering},
  author={Hudson et al. (2019)},
  year={2019},
  note={arXiv:1902.09506}
}

• arXiv: 1902.09506