Iquad V1 Eval

iquad-v1-eval

IQA: Visual Question Answering in Interactive Environments — Gordon et al. (2017) (arXiv:1712.03316, 2017)

What this evaluates

Evaluates an agent's ability to navigate, perceive, and interact with dynamic 3D environments to answer visual questions. It probes spatial reasoning, long-term memory of object locations, and the ability to plan exploration based on question semantics.

Datasets

• iquad v1 — total ?; splits: test (-1); repo https://github.com/danielgordon10/thor-iqa-cvpr-2018

Metrics

• Top-1 question answering accuracy (primary) — range: [0, 1]
  • The proportion of questions for which the model's predicted answer exactly matches the ground truth answer. Calculated as correct predictions divided by total questions.

Input / output format

Input: Egocentric RGB image frames concatenated with object detection channels (YOLO or ground truth) and depth, along with a question string processed by an LSTM.

Output: Action probabilities (π_i) for navigation/interaction actions, a value estimate (v), and a softmax probability distribution over the space of all possible answers. The final output is the argmax answer.

Scoring recipe

correct = 0
for pred, gold in zip(predictions, gold_answers):
    if pred == gold:
        correct += 1
accuracy = correct / len(predictions)

Common pitfalls

• Models without explicit spatial memory or validity loss fail to persist object detections across timesteps, leading to performance worse than random chance.
• The dataset is constructed to eliminate language bias by ensuring each question has a scene configuration for every possible answer, but models may still exploit visual shortcuts if not properly regularized.
• Invalid actions (navigation/interaction failures) are frequent in baselines; ignoring them during training severely hampers exploration and QA accuracy.

Evidence (verbatim from paper)

We evaluate himnon the iquad v1dataset, using Top-1 question answering accuracy.

Citation

@misc{gordon2017iqa,
  title={IQA: Visual Question Answering in Interactive Environments},
  author={Gordon et al. (2017)},
  year={2017},
  note={arXiv:1712.03316}
}

• arXiv: 1712.03316