Roadscapesqa Eval

roadscapesqa-eval

RoadscapesQA: A Multitask, Multimodal Dataset for Visual Question Answering on Indian Roads — Iyer et al. (2026) (arXiv:2602.12877, 2026)

What this evaluates

Evaluates vision-language models on visual question answering for Indian road scenes. It probes capabilities in object counting, object description, and surrounding scene description across diverse driving environments.

Datasets

• RoadscapesQA — total 9000; splits: test (1500); repo https://github.com/vijpandaturtle/roadscapes

Metrics

• exact-match accuracy (primary) — range: [0, 1]
  • Proportion of instances where the model's predicted answer exactly matches the ground truth answer character-for-character.
• cosine similarity — range: other
  • Cosine similarity between sentence embeddings of the model's free-form output and the ground truth answer, computed using the all-MiniLM-L6-v2 embedding model.

Input / output format

Input: Image-question pairs (a single image and a text question)

Output: Free-form text response

Scoring recipe

def compute_metrics(predictions, golds, tasks):
    scores = {}
    for task in ['Object Counting', 'Object Description', 'Surrounding Description']:
        preds = [p for p, t in zip(predictions, tasks) if t == task]
        golds_t = [g for g, t in zip(golds, tasks) if t == task]
        if task == 'Object Counting':
            correct = sum(1 for p, g in zip(preds, golds_t) if p.strip() == str(g))
            scores[task] = correct / len(preds)
        else:
            from sentence_transformers import SentenceTransformer
            import numpy as np
            model = SentenceTransformer('all-MiniLM-L6-v2')
            p_emb = model.encode(preds)
            g_emb = model.encode(golds_t)
            sims = np.sum(p_emb * g_emb, axis=1) / (np.linalg.norm(p_emb, axis=1) * np.linalg.norm(g_emb, axis=1))
            scores[task] = np.mean(sims)
    return scores

Common pitfalls

• Models are evaluated strictly in a zero-shot setting without fine-tuning, which may not reflect their full capability or optimal performance on this domain.
• Exact-match accuracy for object counting is highly sensitive to formatting differences (e.g., 'two' vs '2' or extra whitespace), potentially underestimating semantic correctness.
• Cosine similarity relies on a fixed sentence embedding model (all-MiniLM-L6-v2) that may not capture complex spatial or contextual nuances in driving scenes as effectively as human evaluation or LLM-as-a-judge approaches.

Evidence (verbatim from paper)

For the Object Counting task, we employ exact-match accuracy as the primary evaluation metric, following established practiceChen et al. ([2024]). The Object Description and Surrounding Description tasks are evaluated using cosine similarity between sentence embeddings, specifically utilizing the all-MiniLM-L6-v2 modelWang and others ([2020]).

Citation

@misc{iyer2026roadscapesqa,
  title={RoadscapesQA: A Multitask, Multimodal Dataset for Visual Question Answering on Indian Roads},
  author={Iyer et al. (2026)},
  year={2026},
  note={arXiv:2602.12877}
}

• arXiv: 2602.12877