lrs-vqa-eval
When Large Vision-Language Model Meets Large Remote Sensing Imagery: Coarse-to-Fine Text-Guided Token Pruning — Junwei Luo et al. (2025) (arXiv:2503.07588, 2025)
What this evaluates
Evaluates large vision-language models on high-resolution remote sensing imagery by testing their ability to answer questions about color, count, position, and open-ended descriptions. It probes the model's perception and reasoning capabilities on complex, large-scale satellite/aerial images.
Datasets
- MME-RealWorld-RS — total 1298; splits: test (1298)
- LRS-VQA — total 7333; splits: test (7333)
Metrics
accuracy (primary) — range: percent
- Calculated as the percentage of correctly answered questions. For multiple-choice (color, count, position), exact match is used. For short open-ended answers, a WordNet-based semantic similarity score is computed, and the answer is considered correct if similarity ≥ 0.8.
Input / output format
Input: High-resolution remote sensing image (up to 8064×8064 pixels or dynamically selected tiles) paired with a natural language question.
Output: Natural language text answer (short open-ended or multiple-choice option).
Scoring recipe
def compute_accuracy(predictions, golds):
correct = 0
for pred, gold in zip(predictions, golds):
if pred.strip() == gold.strip():
correct += 1
elif wordnet_similarity(pred, gold) >= 0.8:
correct += 1
return (correct / len(golds)) * 100
Common pitfalls
- Vicuna-1.5-based models tend to respond with only option A; the prompt must be modified by removing 'The best answer is:' to get valid outputs.
- Open-ended answers are not scored via exact string matching; they require WordNet-based semantic similarity (threshold ≥ 0.8).
- Localization recall (Table 7) only counts as successful if retained vision tokens cover >50% of the ground-truth region.
Evidence (verbatim from paper)
Benchmarks and evaluation metrics: i) MME-RealWorld-RS: the RS part of MME-Realworld[[91]], containing 1,298 RSIs with expert-annotated questions in three types: color, count, and position. We follow the official evaluation script but modify the prompt by removing “The best answer is:" to address Vicuna-1.5-based models’ tendency to respond with only option A. ii) LRS-VQA: it consists of 3 parts: LRS-FAIR, LRS-Bridge, and LRS-STAR, containing 2,272, 1,062, and 3,999 QA pairs, respectively. For the short open-ended format, we adopt a structured evaluation metric following[[22], [71]], using WordNet[[52]], with a semantic similarity threshold of 0.8.
Citation
@misc{luo2025lrs_vqa,
title={When Large Vision-Language Model Meets Large Remote Sensing Imagery: Coarse-to-Fine Text-Guided Token Pruning},
author={Junwei Luo et al. (2025)},
year={2025},
note={arXiv:2503.07588}
}
1---2name: lrs-vqa-eval3description: Evaluates large vision-language models on high-resolution remote sensing imagery by testing their ability to answer questions about color, count, position, and open-ended descriptions. It probes the model's perception and reasoning capabilities on complex, large-scale satellite/aerial images. Use when the user wants to benchmark on MME-RealWorld-RS, LRS-VQA, or asks about evaluating this task. Reports accuracy.4---56# lrs-vqa-eval78> When Large Vision-Language Model Meets Large Remote Sensing Imagery: Coarse-to-Fine Text-Guided Token Pruning — Junwei Luo et al. (2025) (arXiv:2503.07588, 2025)910## What this evaluates1112Evaluates large vision-language models on high-resolution remote sensing imagery by testing their ability to answer questions about color, count, position, and open-ended descriptions. It probes the model's perception and reasoning capabilities on complex, large-scale satellite/aerial images.1314## Datasets1516- **MME-RealWorld-RS** — total 1298; splits: test (1298)17- **LRS-VQA** — total 7333; splits: test (7333)1819## Metrics2021- `accuracy` **(primary)** — range: percent22 - Calculated as the percentage of correctly answered questions. For multiple-choice (color, count, position), exact match is used. For short open-ended answers, a WordNet-based semantic similarity score is computed, and the answer is considered correct if similarity ≥ 0.8.2324## Input / output format2526**Input**: High-resolution remote sensing image (up to 8064×8064 pixels or dynamically selected tiles) paired with a natural language question.2728**Output**: Natural language text answer (short open-ended or multiple-choice option).2930## Scoring recipe3132```python33def compute_accuracy(predictions, golds):34 correct = 035 for pred, gold in zip(predictions, golds):36 if pred.strip() == gold.strip():37 correct += 138 elif wordnet_similarity(pred, gold) >= 0.8:39 correct += 140 return (correct / len(golds)) * 10041```4243## Common pitfalls4445- Vicuna-1.5-based models tend to respond with only option A; the prompt must be modified by removing 'The best answer is:' to get valid outputs.46- Open-ended answers are not scored via exact string matching; they require WordNet-based semantic similarity (threshold ≥ 0.8).47- Localization recall (Table 7) only counts as successful if retained vision tokens cover >50% of the ground-truth region.4849## Evidence (verbatim from paper)5051> Benchmarks and evaluation metrics: i) MME-RealWorld-RS: the RS part of MME-Realworld[[91]], containing 1,298 RSIs with expert-annotated questions in three types: color, count, and position. We follow the official evaluation script but modify the prompt by removing “The best answer is:" to address Vicuna-1.5-based models’ tendency to respond with only option A. ii) LRS-VQA: it consists of 3 parts: LRS-FAIR, LRS-Bridge, and LRS-STAR, containing 2,272, 1,062, and 3,999 QA pairs, respectively. For the short open-ended format, we adopt a structured evaluation metric following[[22], [71]], using WordNet[[52]], with a semantic similarity threshold of 0.8.5253## Citation5455```bibtex56@misc{luo2025lrs_vqa,57 title={When Large Vision-Language Model Meets Large Remote Sensing Imagery: Coarse-to-Fine Text-Guided Token Pruning},58 author={Junwei Luo et al. (2025)},59 year={2025},60 note={arXiv:2503.07588}61}62```6364- arXiv: 2503.07588
qhjqhj00