Robusto 1 Eval

robusto-1-eval

Robusto-1 Dataset: Comparing Humans and VLMs on real out-of-distribution Autonomous Driving VQA from Peru — Dunant Cusipuma et al. (2025) (arXiv:2503.07587, 2025)

What this evaluates

Evaluates the cognitive alignment and visuocognitive reasoning of Vision-Language Models (VLMs) compared to humans on real-world, out-of-distribution autonomous driving scenarios from Peru. It probes how models and humans interpret complex, rare driving situations through open-ended, multiple-choice, and counterfactual/hypothetical visual question answering.

Datasets

• Robusto-1 — total ?; splits: test (-1)

Metrics

• Representational Similarity Analysis (RSA) (primary) — range: other
  • Computes similarity matrices using Gramian matrices of system response embeddings. Measures cognitive alignment by calculating the geometric distance between each system's response embedding and the median response across all systems per question.

Input / output format

Input: A driving video clip paired with a natural language question (open-ended, multiple-choice, or counterfactual/hypothetical).

Output: Natural language response or selected option from the model/human.

Scoring recipe

def evaluate_responses(responses_per_question):
    embeddings = [encode(r) for r in responses_per_question]
    median_emb = np.median(embeddings, axis=0)
    distances = [cosine_dist(e, median_emb) for e in embeddings]
    similarity_matrix = np.dot(embeddings, embeddings.T)  # Gramian
    return distances, similarity_matrix

Common pitfalls

• Assuming high alignment in 2D PCA projections indicates true cognitive similarity, as the paper explicitly warns this is misleading due to low explained variance.
• Overlooking question-type bias: Multiple-choice questions naturally show higher convergence due to limited answer space, which can mask underlying reasoning differences.

Evidence (verbatim from paper)

We first begin by analyzing the results from computing the similarity matrices through the Gramian matrices of each system as done in RSA... To better understand at a more fine-grained level how each system’s answer geometrically steered the overall system’s topology for the similarity matrices, we proceeded to perform an inter-system agreement analysis by computing the distances between each embedding to the median response per answer across all systems for every question and answer pair.

Citation

@misc{cusipuma2025robusto1,
  title={Robusto-1 Dataset: Comparing Humans and VLMs on real out-of-distribution Autonomous Driving VQA from Peru},
  author={Dunant Cusipuma et al. (2025)},
  year={2025},
  note={arXiv:2503.07587}
}

• arXiv: 2503.07587