Xlrs Bench Eval

xlrs-bench-eval

GeoLLaVA-8K: Scaling Remote-Sensing Multimodal Large Language Models to 8K Resolution — Wang et al. (2025) (arXiv:2505.21375, 2025)

What this evaluates

Evaluates multimodal large language models on ultra-high-resolution remote sensing imagery using vision-language question answering. It probes both perception (e.g., object classification, counting, spatial relations) and reasoning capabilities across various sub-tasks.

Datasets

• XLRS-Bench — total ?; splits: test (-1)
• LRS-VQA — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: percent
  • Percentage of correctly answered questions. For XLRS-Bench, accuracy is computed per sub-task (L-1 dimension) and averaged across all sub-tasks. For LRS-VQA, accuracy is computed after converting open-ended QA to multiple-choice format.

Input / output format

Input: Ultra-high-resolution remote sensing images (up to 8K resolution) paired with zero-shot, uniform text prompts/questions.

Output: Text responses to VQA questions. For LRS-VQA, responses are matched against multiple-choice options generated by GPT-4o.

Scoring recipe

def compute_accuracy(predictions, gold_labels):
    correct = 0
    for pred, gold in zip(predictions, gold_labels):
        if normalize_text(pred) == normalize_text(gold):
            correct += 1
    return (correct / len(gold_labels)) * 100

Common pitfalls

• The paper explicitly reports only the L-1 dimension accuracy for XLRS-Bench; L-3 and L-4 results are in the appendix and should not be used as the primary metric.
• LRS-VQA evaluation requires converting open-ended answers to multiple-choice using an external LLM (GPT-4o), which may introduce format bias or hallucination in distractor generation.
• All models must be evaluated in a zero-shot setting with uniform prompts to ensure fair comparison.

Evidence (verbatim from paper)

Following XLRS-Bench*[xlrs-bench], we evaluated the accuracy and reported of L-1 dimension for the VQA task, with L-3 and L-4 results available in the appendix. GeoLLaVA-8K achieved 56.28% accuracy, outperforming LLaVA-Next[llava-next]* (55.07%) by 1.21%, confirming strong generalization to new datasets and formats.

Citation

@misc{wang2025geollava8k,
  title={GeoLLaVA-8K: Scaling Remote-Sensing Multimodal Large Language Models to 8K Resolution},
  author={Wang et al. (2025)},
  year={2025},
  note={arXiv:2505.21375}
}

• arXiv: 2505.21375