Hvqr Eval

hvqr-eval

Explainable High-order Visual Question Reasoning: A New Benchmark and Knowledge-routed Network — Cao et al. (2019) (arXiv:1909.10128, 2019)

What this evaluates

This benchmark evaluates a model's ability to perform high-order, multistep visual question answering by integrating visual scene graphs with external commonsense knowledge. It explicitly probes the model's reasoning process by requiring it to predict intermediate knowledge triplets alongside the final answer, enforcing explainability and self-diagnosis capabilities.

Datasets

• HVQR — total 157201; splits: train (94815), val (30676), test (31710)

Metrics

• triplet recall (primary) — range: [0, 1]
  • For each question, recall is calculated as the number of correctly predicted reasoning triplets divided by the number of predicted triplets. The final metric is the average recall across all QA pairs in the dataset.

Input / output format

Input: A natural image and a natural language question requiring multistep reasoning.

Output: A final answer string and a sequence of predicted reasoning triplets (or query layout) representing the intermediate inference steps.

Scoring recipe

def compute_triplet_recall(predictions, golds):
    recalls = []
    for pred_triplets, gold_triplets in zip(predictions, golds):
        if len(pred_triplets) == 0:
            recalls.append(0.0)
        else:
            correct = sum(1 for t in pred_triplets if t in gold_triplets)
            recalls.append(correct / len(pred_triplets))
    return sum(recalls) / len(recalls)

Common pitfalls

• The dataset enforces that each knowledge triplet appears only once across all questions to prevent overfitting; models relying on memorized facts will fail on test splits.
• Evaluation requires explicit prediction of intermediate reasoning triplets, not just the final answer; black-box VQA models that skip step-by-step reasoning will score zero on the primary metric.
• Questions are human-rewritten for diversity and readability, so template-matching or rigid parsing strategies will not work.

Evidence (verbatim from paper)

For each QA pair, the explainable evaluation metric calculates triplet precisions for each question, $recall=\frac{#correct_triplets}{#predicted_triplets}$ and the average recall of all QA pairs as the final recall.

Citation

@misc{cao2019hvqr,
  title={Explainable High-order Visual Question Reasoning: A New Benchmark and Knowledge-routed Network},
  author={Cao et al. (2019)},
  year={2019},
  note={arXiv:1909.10128}
}

• arXiv: 1909.10128