Bigearthnet Txt Eval

bigearthnet-txt-eval

BigEarthNet.txt: A Large-Scale Multi-Sensor Image-Text Dataset and Benchmark for Earth Observation — Herzog et al. (2026) (arXiv:2603.29630, 2026)

What this evaluates

Evaluates vision-language models on remote sensing tasks including image captioning, binary visual question answering, multiple-choice questions, and referring expression/point detection. It probes the models' ability to understand multi-sensor (SAR + multispectral) and RGB Earth observation imagery, follow complex spatial instructions, and generate accurate land-use/land-cover descriptions or localized bounding boxes.

Datasets

• BigEarthNet.txt — total 464044; splits: benchmark (-1)

Metrics

• accuracy (primary) — range: percent
  • Percentage of correctly answered questions for binary VQA and MCQ tasks. Calculated as the number of exact matches between predicted and gold answers divided by total instances.
• mIoU — range: percent
  • Mean intersection-over-union between predicted and reference bounding boxes or segmentation masks for referring expression and point detection tasks.
• BLEU-4 — range: percent
  • N-gram-based metric measuring the precision of 4-gram overlaps between generated captions and reference captions, with a brevity penalty.
• CLAIR — range: percent
  • LLM-based metric where a judge model (DeepSeek-R1-Distill-Qwen-32B) outputs a score between 0 and 100 indicating how likely the candidate caption describes the same image as the reference.

Input / output format

Input: Multi-sensor (Sentinel-1 SAR + Sentinel-2 multispectral) or RGB (Sentinel-2 bands) Earth observation images paired with task-specific instructions (e.g., captioning prompts, binary/MCQ questions, referring expressions with or without a spatial point prior).

Output: Text responses (captions, yes/no answers, selected options, or bounding box coordinates) extracted to match the specified format; unambiguous answers are required for instruction-following tracking.

Scoring recipe

def evaluate(predictions, golds, task_type):
    # Extract answers even if they do not strictly adhere to the specified format
    preds_clean = extract_unambiguous_answers(predictions)
    if task_type in ('vqa', 'mcq'):
        return sum(1 for p, g in zip(preds_clean, golds) if p == g) / len(golds) * 100
    elif task_type == 'referring_detection':
        ious = [iou(p, g) for p, g in zip(preds_clean, golds)]
        mIoU = sum(ious) / len(ious) * 100
        acc_k = {k: sum(1 for i in ious if i >= k/100) / len(ious) * 100 for k in [25, 50, 75, 90]}
        return mIoU, acc_k
    elif task_type == 'captioning':
        return compute_ngram_embedding_metrics(preds_clean, golds)
    elif task_type == 'captioning_clair':
        return llm_judge_score(preds_clean, golds)

Common pitfalls

• Models accepting multispectral/multi-sensor inputs often perform worse than RGB-only versions because they were pre-trained/fine-tuned on RGB data and lack inference-time adaptation to exploit extra spectral bands.
• Instruction-following is a major bottleneck; many models fail to consistently adhere to MCQ or binary VQA formats, requiring careful post-hoc answer extraction to compute accuracy.
• Referring expression detection without a point prior is significantly harder than with a point prior, as the latter reduces the task to local boundary estimation rather than open-vocabulary instance search.

Evidence (verbatim from paper)

Reported metrics for captioning: BLEU-4, binary VQA (visual question answering): accuracy, MCQ (multiple-choice question): accuracy, and referring expression detection: mIoU (mean intersection-over-union). All results in percent (%).

Citation

@misc{herzog2026bigearthnettxt,
  title={BigEarthNet.txt: A Large-Scale Multi-Sensor Image-Text Dataset and Benchmark for Earth Observation},
  author={Herzog et al. (2026)},
  year={2026},
  note={arXiv:2603.29630}
}

• arXiv: 2603.29630