Forest Change Eval

forest-change-eval

Forest-Chat: Adapting Vision-Language Agents for Interactive Forest Change Analysis — Brock et al. (2026) (arXiv:2601.14637, 2026)

What this evaluates

Evaluates a vision-language agent's ability to perform joint change detection and natural language captioning on bi-temporal remote sensing imagery. It probes the model's capacity to segment deforestation and built-environment changes at the pixel level while generating accurate semantic descriptions of those changes.

Datasets

• Forest-Change — total ?; splits: test (-1); repo https://github.com/JamesBrockUoB/ForestChat
• LEVIR-MCI-Trees — total ?; splits: test (-1)

Metrics

• MIoU (primary) — range: [0, 1]
  • Mean Intersection over Union across all pixels and classes. Computed as the average of IoU per class: IoU = intersection(pred, gt) / union(pred, gt).
• BLEU-4 — range: [0, 1]
  • 4-gram BLEU score measuring n-gram precision with a brevity penalty. Standard machine translation metric for caption quality.
• CIDEr-D — range: [0, 1]
  • Consensus-based Image Description Evaluation metric that weights n-grams by their IDF scores across the dataset.
• METEOR — range: [0, 1]
  • Metric for Evaluating Translation that explicitly models synonymy and stem matching, combining precision and recall.
• ROUGEL — range: [0, 1]
  • Longest Common Subsequence-based ROUGE metric that captures sentence-level fluency and recall.

Input / output format

Input: Paired bi-temporal remote sensing images (before and after change) with corresponding ground-truth segmentation masks and reference captions.

Output: Pixel-level binary or multi-class change masks and a single natural language caption describing the detected changes.

Scoring recipe

def evaluate(pred_masks, gt_masks, pred_captions, gt_captions):
    miou = np.mean([np.sum((p==g)&(p==1))/np.sum((p==g)|(p==1)) for p,g in zip(pred_masks, gt_masks)])
    bleu4 = nltk.translate.bleu_score.sentence_bleu(gt_captions, pred_captions, weights=(0.25,0.25,0.25,0.25))
    meteor = compute_meteor(pred_captions, gt_captions)
    rougeL = compute_rougeL(pred_captions, gt_captions)
    cider = compute_cider(pred_captions, gt_captions)
    return {'MIoU': miou, 'BLEU-4': bleu4, 'METEOR': meteor, 'ROUGEL': rougeL, 'CIDEr-D': cider}

Common pitfalls

• Class imbalance and small patch sizes (<400 pixels) in Forest-Change cause models to miss deforestation changes, lowering IoU scores.
• Fuzzy boundaries at deforestation edges reduce pixel-level overlap metrics compared to crisp urban changes.
• Atmospheric artifacts and seasonal variations trigger false positives in zero-shot inference, penalizing MIoU.
• Captioning metric scores (BLEU, METEOR) vary significantly due to differing annotation styles between Forest-Change and LEVIR-MCI-Trees.

Evidence (verbatim from paper)

Captioning performance evaluation is more nuanced. For the LEVIR-MCI-Trees dataset, the BLEU-1, BLEU-2, and CIDEr-D metrics are generally higher, while BLEU-3, BLEU-4, METEOR, and ROUGEL are better for Forest-Change.

Citation

@misc{brock2026forestchat,
  title={Forest-Chat: Adapting Vision-Language Agents for Interactive Forest Change Analysis},
  author={Brock et al. (2026)},
  year={2026},
  note={arXiv:2601.14637}
}

• arXiv: 2601.14637