Earthvlset Eval

earthvlset-eval

EarthVL: A Progressive Earth Vision-Language Understanding and Generation Framework — Junjue Wang et al. (2026) (arXiv:2601.02783, 2026)

What this evaluates

Evaluates high-spatial-resolution remote sensing models on land-cover semantic segmentation and visual question answering. It probes pixel-level object recognition, spatial reasoning, and relational counting capabilities in complex urban scenes.

Datasets

• EarthVLSet — total 11295; splits: train (5260), val (2699), test (3336); repo https://github.com/Junjue-Wang/EarthVL

Metrics

• mIoU (primary) — range: percent
  • Mean Intersection over Union across all land-cover categories. Calculated as the average of IoU (intersection over union) for each class.
• OA (primary) — range: percent
  • Overall Accuracy for multiple-choice VQA, calculated as the percentage of correctly predicted answers out of total questions.
• RMSE — range: other
  • Root Mean Square Error used specifically to evaluate counting tasks in VQA. Lower values indicate better counting accuracy.

Input / output format

Input: High-spatial-resolution remote sensing images. For VQA, images paired with multiple-choice or open-ended questions. Segmentation-guided models also receive pixel-level semantic masks or features.

Output: For segmentation: pixel-wise class labels or probability maps. For VQA: a selected multiple-choice option or a generated text response.

Scoring recipe

def compute_miou(pred_masks, gt_masks, num_classes):
    ious = []
    for c in range(num_classes):
        intersection = np.sum((pred_masks == c) & (gt_masks == c))
        union = np.sum((pred_masks == c) | (gt_masks == c))
        ious.append(intersection / union if union > 0 else 0)
    return np.mean(ious) * 100

def compute_oa(predictions, gold_answers):
    correct = sum(1 for p, g in zip(predictions, gold_answers) if p == g)
    return (correct / len(gold_answers)) * 100

def compute_rmse(predictions, gold_counts):
    return np.sqrt(np.mean((np.array(predictions) - np.array(gold_counts))**2))

Common pitfalls

• General-purpose VLMs suffer from a significant domain gap on remote sensing imagery, leading to poor zero-shot accuracy.
• Counting and complex relational questions heavily depend on pixel-level segmentation guidance; models without segmentation features show significantly higher RMSE.
• Multiple-choice VQA performance is more sensitive to vision encoder quality than language model size, contrary to typical LLM benchmarks.

Evidence (verbatim from paper)

Following the common settings, we adopted the classification accuracy and root-mean-square error (RMSE) as the evaluation metrics, with the RMSE used to evaluate the counting tasks.

Citation

@misc{wang2026earthvl,
  title={EarthVL: A Progressive Earth Vision-Language Understanding and Generation Framework},
  author={Junjue Wang et al. (2026)},
  year={2026},
  note={arXiv:2601.02783}
}

• arXiv: 2601.02783