Rscc Change Caption Eval

rscc-change-caption-eval

RSCC: A Large-Scale Remote Sensing Change Caption Dataset for Disaster Events — Chen et al. (2025) (arXiv:2509.01907, 2025)

What this evaluates

Evaluates vision-language models' ability to generate detailed, semantically accurate captions describing changes between bi-temporal remote sensing image pairs, particularly in disaster scenarios. It probes spatiotemporal reasoning, fine-grained environmental change detection, and long-text generation quality.

Datasets

• RSCC — total 62351; splits: test (-1); repo https://github.com/Bili-Sakura/RSCC

Metrics

• ST5-SCS (primary) — range: percent
  • Sentence T5-XXL Embedding with Sharpened Cosine Similarity. Computes cosine similarity between T5-XXL embeddings of the reference and hypothesis captions, applying a sharpening function with parameters q=0 and p=3.
• ROUGE — range: percent
  • Standard n-gram overlap metric measuring recall/precision of n-grams between reference and generated captions.
• METEOR — range: percent
  • N-gram overlap metric that incorporates synonymy, stemming, and word order to measure semantic similarity.

Input / output format

Input: Bi-temporal remote sensing image pairs (pre- and post-disaster). Optionally augmented with building damage masks/info from the xBD dataset.

Output: Natural language caption describing the observed changes between the two images.

Scoring recipe

def compute_st5_scs(predictions, references, q=0, p=3):
    ref_embeds = get_t5_xxl_embedding(references)
    pred_embeds = get_t5_xxl_embedding(predictions)
    cos_sim = cosine_similarity(ref_embeds, pred_embeds)
    sharpened = (cos_sim + q) ** p
    return np.mean(sharpened) * 100

Common pitfalls

• N-gram overlap metrics (ROUGE, METEOR) are suboptimal for measuring semantic similarity in long captions.
• Some models suffer from severe output repetition, artificially inflating caption length and degrading scores.
• Scaling correction decoding strategies (VCD, DoLa, DeCo) do not consistently improve performance on this complex spatiotemporal reasoning task.

Evidence (verbatim from paper)

For model evaluation, we compare the text similarity with n-gram overlap metrics including ROUGE and METEOR. While the aforementioned measures are commonly reported in image captioning works, we find they are suboptimal to measure the semantic similarity across long texts. Therefore, we follow Kaggle LLM Prompt Recovery Competition and introduce Sentence T5-XXL Embedding with Sharpened Cosine Similarity (ST5-SCS) to get a well-established similarity measure. We set q=0 and p=3 for sharpened cosine similarity.

Citation

@misc{chen2025rscc,
  title={RSCC: A Large-Scale Remote Sensing Change Caption Dataset for Disaster Events},
  author={Chen et al. (2025)},
  year={2025},
  note={arXiv:2509.01907}
}

• arXiv: 2509.01907