Groundset Eval

groundset-eval

GroundSet: A Cadastral-Grounded Dataset for Spatial Understanding with Vector Data — Ferrod et al. (2026) (arXiv:2603.14609, 2026)

What this evaluates

Probes zero-shot spatial understanding and grounding capabilities of multimodal LLMs on high-resolution remote sensing imagery. Evaluates generalization across captioning, classification, detection, segmentation, and VQA tasks using verified cadastral vector annotations.

Datasets

• GroundSet — total ?; splits: test (-1)

Metrics

• CIDEr — range: [0, 1] or percent
  • Consensus-based Image Description Evaluation metric for captioning quality.
• Acc@0.8 — range: [0, 1]
  • Accuracy based on semantic similarity threshold k=0.8 using all-mpnet-base-v2 embeddings after LLM extraction and normalization.
• F1@0.5 (primary) — range: [0, 1]
  • F1 score computed with IoU threshold 0.5. TP: IoU≥0.5 with unclaimed GT; FP: IoU<0.5 or duplicate; FN: unmatched GT; TN: correct rejection. Taxonomy-aware (child class predicts parent correctly).
• mIoU — range: [0, 1]
  • Mean Intersection over Union across output modalities (HBB, OBB, mask) regardless of format.

Input / output format

Input: High-resolution remote sensing image paired with a task-specific prompt (e.g., conversational instruction, task prefix like 'detect [object]', or referring expression). Prompts are adapted per model architecture to ensure fair zero-shot assessment.

Output: Natural language response, bounding box coordinates (HBB or OBB), segmentation mask, or class label. Predictions are geometrically recovered (e.g., OBB to HBB, coordinate rescaling) prior to scoring.

Scoring recipe

def compute_f1_at_05(preds, gts):
    ious = [compute_iou(p, g) for p, g in zip(preds, gts)]
    tp = sum(1 for iou in ious if iou >= 0.5)
    fp = sum(1 for iou in ious if iou < 0.5) + count_duplicates(preds)
    fn = count_unmatched(gts)
    prec = tp / (tp + fp) if (tp + fp) > 0 else 0
    rec = tp / (tp + fn) if (tp + fn) > 0 else 0
    return 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0

Common pitfalls

• Formatting deviations in model outputs (e.g., conversational filler, coordinate norms) are not penalized; geometric recovery is required before scoring.
• Taxonomy-aware evaluation: predicting a valid child class for a parent class query counts as correct, which may inflate scores if not accounted for.
• Model-specific prompt adaptation is necessary; using standard conversational prompts yields poor zero-shot performance for some architectures (e.g., PaliGemma requires task prefixes).

Evidence (verbatim from paper)

For Object Detection, Multi-Class Detection, Referring Expression Comprehension and Segmentation, we compute mIoU regardless of the output modality (HBB, OBB or mask). Crucially, evaluation is taxonomy-aware: predicting a valid child class (e.g., Church) when queried for a parent class (e.g., Building) is considered correct. We select the candidate with maximum IoU per target and report F1, Precision and Recall based on: TP: IoU ≥ 0.5 with an unclaimed ground truth. FP: IoU <0.5 or duplicate prediction for the same ground truth. FN: Unmatched ground truth object. TN: Correct rejection (no object predicted when none exists).

Citation

@misc{ferrod2026groundset,
  title={GroundSet: A Cadastral-Grounded Dataset for Spatial Understanding with Vector Data},
  author={Ferrod et al. (2026)},
  year={2026},
  note={arXiv:2603.14609}
}

• arXiv: 2603.14609