Vsdx Eval

vsdx-eval

Enhanced Vision-Language Models for Diverse Sensor Understanding: Cost-Efficient Optimization and Benchmarking — Sangyun Chung et al. (2024) (arXiv:2412.20750, 2024)

What this evaluates

Evaluates vision-language models' ability to perceive and reason about non-RGB sensor data (thermal, depth, X-ray). It probes low-level perception (existence, counting, position, description) and high-level understanding (contextual reasoning, sensor-specific physical property interpretation).

Datasets

• VS-TDX — total 10160; splits: test (10160)

Metrics

• accuracy (primary) — range: percent
  • Percentage of question-answer pairs where the model's generated response matches the ground-truth positive answer. Scores are reported per sub-task (Existence, Counting, Position, General Description, Contextual Understanding, Sensor Understanding) and aggregated into Perception and Understanding averages.

Input / output format

Input: A single image (thermal, depth, or X-ray modality) paired with a natural language question.

Output: A natural language text response answering the question.

Scoring recipe

correct = 0
total = 0
for qa in dataset:
    pred = model.generate(qa.image, qa.question)
    if normalize(pred) == normalize(qa.gold):
        correct += 1
    total += 1
accuracy = (correct / total) * 100

Common pitfalls

• Models are evaluated on non-RGB modalities that lack color cues, so RGB-biased priors often lead to systematic misinterpretations of physical properties.
• Sub-task scores (e.g., Counting, Position) are averaged to form the Perception and Understanding aggregate metrics rather than being combined into a single global accuracy score.
• The benchmark includes a negative answer set used for training (DNA optimization), but evaluation accuracy is computed solely against the positive answer set.

Evidence (verbatim from paper)

The final VS-TDX benchmark comprises approximately 6,791 sensor images, consisting of 1,867 thermal, 2,781 depth, and 2,143 X-ray images. It includes 10,160 diverse question-answer pairs, with an average of 1.50 pairs per image. TABLE I: Evaluation results of various Vision-Language Models (VLMs) on the VS-TDX benchmark, with accuracy as the primary metric.

Citation

@misc{chung2024enhanced,
  title={Enhanced Vision-Language Models for Diverse Sensor Understanding: Cost-Efficient Optimization and Benchmarking},
  author={Sangyun Chung et al. (2024)},
  year={2024},
  note={arXiv:2412.20750}
}

• arXiv: 2412.20750