Rsrcc Eval

rsrcc-eval

RSRCC: A Remote Sensing Regional Change Comprehension Benchmark Constructed via Retrieval-Augmented Best-of-N Ranking — Kazoom et al. (2026) (arXiv:2604.20623, 2026)

What this evaluates

This benchmark evaluates large language models' ability to perform fine-grained, region-specific semantic reasoning on remote sensing image pairs. It probes localized change comprehension by asking models to answer binary, multiple-choice, and open-ended questions about specific changes (e.g., new construction, vegetation loss) within satellite imagery.

Datasets

• RSRCC — total ?; splits: test (-1)

Metrics

• Human Agreement (%) — range: percent
  • Percentage of human evaluators who agree with the generated questions or answers.
• Accuracy (%) (primary) — range: percent
  • Correctness rate for binary (Yes/No) and multiple-choice responses.
• BLEU — range: percent
  • n-gram precision measuring lexical overlap between model-generated answers and ground-truth answers.
• BERTScore (F1) — range: [0, 1]
  • Semantic similarity score between model-generated and human-written captions using contextual embeddings.
• CIDEr — range: percent
  • Consensus-based metric measuring similarity between generated captions and multiple human references using TF-IDF weighted n-grams.
• SPICE — range: percent
  • Semantic propositional metric evaluating agreement between generated and reference captions at the level of objects, attributes, and relations.

Input / output format

Input: Paired high-resolution satellite images (typically 512x512 px) with a region-specific question (Yes/No, multiple-choice, or open-ended) targeting localized changes.

Output: For closed-ended questions: 'Yes'/'No' or a selected option. For open-ended questions: a free-text caption or answer.

Scoring recipe

if question_type in ['Yes/No', 'Multiple-Choice']:
    score = 1.0 if prediction == gold else 0.0
    return mean(score)
elif question_type == 'Open-Ended':
    bleu = compute_bleu(prediction, gold)
    bertscore = compute_bertscore_f1(prediction, gold)
    cider = compute_cider(prediction, gold_references)
    spice = compute_spice(prediction, gold_references)
    return {'BLEU': bleu, 'BERTScore': bertscore, 'CIDEr': cider, 'SPICE': spice}

Common pitfalls

• Accuracy is only applicable to closed-ended (Yes/No, Multiple-Choice) questions, not open-ended ones.
• BLEU, CIDEr, and SPICE are strictly for open-ended responses and should not be computed for binary/multiple-choice outputs.
• Human Agreement is evaluated per dataset creation pipeline stage, not just as a final model benchmark metric.

Evidence (verbatim from paper)

We evaluate our framework both quantitatively and qualitatively to assess scalability, reliability, and semantic consistency using the following metrics: Human Agreement (%) - percentage of evaluators agreeing with generated questions or answers. Accuracy (%) - correctness of binary (Yes/No) and multiple-choice responses. BLEU - $n$-gram precision measuring lexical overlap between model-generated answers and the ground-truth answers in the dataset. BERTScore (F1) - semantic similarity between model-generated and human-written captions.

Citation

@misc{kazoom2026rsrcc,
  title={RSRCC: A Remote Sensing Regional Change Comprehension Benchmark Constructed via Retrieval-Augmented Best-of-N Ranking},
  author={Kazoom et al. (2026)},
  year={2026},
  note={arXiv:2604.20623}
}

• arXiv: 2604.20623