Tdbench Eval

tdbench-eval

TDBench: A Benchmark for Top-Down Image Understanding with Reliability Analysis of Vision-Language Models — Hou et al. (2025) (arXiv:2504.03748, 2025)

What this evaluates

Evaluates vision-language models on top-down (aerial) image understanding by testing their ability to answer questions about rotated views. It measures rotational consistency to filter out hallucinations and decomposes performance into true knowledge versus lucky guessing via a probabilistic reliability framework.

Datasets

• TDBench — total ?; splits: test (-1); repo https://github.com/Columbia-ICSL/TDBench

Metrics

• RotationalEval (RE) (primary) — range: [0, 1]
  • Probability that a model answers correctly across all four rotations (0°, 90°, 180°, 270°) of the same image-question pair. RE = Pr(∀φ∈Φ: Y_i^(φ)=1).
• VanillaEval (VE) — range: [0, 1]
  • Single-pass accuracy on the original (0°) image, or average accuracy across four rotations. VE@0° is used for baseline comparison.
• Adjusted Accuracy (A_adj) — range: [0, 1]
  • Decomposes performance into true knowledge (θ) and accuracy on known questions (r). Computed as A_adj = θ * r, where θ and r are inferred by solving a system of equations based on RE, average VE, and MA (all-rotations-wrong rate).

Input / output format

Input: Top-down/aerial images paired with questions. Each image is presented in four orientations (0°, 90°, 180°, 270°) for rotational consistency testing.

Output: Text responses answering the questions.

Scoring recipe

# For each question i and rotation phi in [0, 90, 180, 270]:
#   Y_i_phi = 1 if model_answer == gold_answer else 0
# RE = mean(Y_i_0 * Y_i_90 * Y_i_180 * Y_i_270 for all i)
# VE_bar = mean(mean(Y_i_phi for phi in rotations) for all i)
# MA = mean(1 - max(Y_i_phi for phi in rotations) for all i)
# Solve system for theta, r, g:
#   RE = theta * r^4 + (1-theta) * g^4
#   VE_bar = theta * r + (1-theta) * g
#   MA = theta * (1-r)^4 + (1-theta) * (1-g)^4
# A_adj = theta * r

Common pitfalls

• Models may achieve high single-pass accuracy (VE) through lucky guesses rather than true visual grounding.
• Chain-of-thought prompting can improve semantic reasoning but may reduce visual grounding, lowering RE scores.
• Rotational consistency requires correct answers across all four orientations; failing one rotation counts as a failure for RE.

Evidence (verbatim from paper)

We define three observations RE = Pr(∀φ∈Φ:Y_i^(φ)=1), VE_bar = E[1/|Φ| Σ Y_i^(φ)], MA = Pr(∀φ:Y_i^(φ)=0)... Assuming each question for the model is either “known” or “unknown”, and rotations are conditionally independent, the above observations satisfy RE = θr^4 + (1-θ)g^4, VE_bar = θr + (1-θ)g, MA = θ(1-r)^4 + (1-θ)(1-g)^4... We aggregate these into the adjusted accuracy (A_adj): A_adj = θ·r.

Citation

@misc{hou2025tdbench,
  title={TDBench: A Benchmark for Top-Down Image Understanding with Reliability Analysis of Vision-Language Models},
  author={Hou et al. (2025)},
  year={2025},
  note={arXiv:2504.03748}
}

• arXiv: 2504.03748