# Groundset Eval

> Probes zero-shot spatial understanding and grounding capabilities of multimodal LLMs on high-resolution remote sensing imagery. Evaluates generalization across captioning, classification, detection, segmentation, and VQA tasks using verified cadastral vector annotations. Use when the user wants to benchmark on GroundSet, or asks about evaluating this task. Reports F1@0.5.

- Skill: `qhjqhj00/groundset-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/groundset-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/groundset-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Productivity
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/groundset-eval

---


# groundset-eval

> GroundSet: A Cadastral-Grounded Dataset for Spatial Understanding with Vector Data — Ferrod et al. (2026) (arXiv:2603.14609, 2026)

## What this evaluates

Probes zero-shot spatial understanding and grounding capabilities of multimodal LLMs on high-resolution remote sensing imagery. Evaluates generalization across captioning, classification, detection, segmentation, and VQA tasks using verified cadastral vector annotations.

## Datasets

- **GroundSet** — total ?; splits: test (-1)

## Metrics

- `CIDEr` — range: [0, 1] or percent
  - Consensus-based Image Description Evaluation metric for captioning quality.
- `Acc@0.8` — range: [0, 1]
  - Accuracy based on semantic similarity threshold k=0.8 using all-mpnet-base-v2 embeddings after LLM extraction and normalization.
- `F1@0.5` **(primary)** — range: [0, 1]
  - F1 score computed with IoU threshold 0.5. TP: IoU≥0.5 with unclaimed GT; FP: IoU<0.5 or duplicate; FN: unmatched GT; TN: correct rejection. Taxonomy-aware (child class predicts parent correctly).
- `mIoU` — range: [0, 1]
  - Mean Intersection over Union across output modalities (HBB, OBB, mask) regardless of format.

## Input / output format

**Input**: High-resolution remote sensing image paired with a task-specific prompt (e.g., conversational instruction, task prefix like 'detect [object]', or referring expression). Prompts are adapted per model architecture to ensure fair zero-shot assessment.

**Output**: Natural language response, bounding box coordinates (HBB or OBB), segmentation mask, or class label. Predictions are geometrically recovered (e.g., OBB to HBB, coordinate rescaling) prior to scoring.

## Scoring recipe

```python
def compute_f1_at_05(preds, gts):
    ious = [compute_iou(p, g) for p, g in zip(preds, gts)]
    tp = sum(1 for iou in ious if iou >= 0.5)
    fp = sum(1 for iou in ious if iou < 0.5) + count_duplicates(preds)
    fn = count_unmatched(gts)
    prec = tp / (tp + fp) if (tp + fp) > 0 else 0
    rec = tp / (tp + fn) if (tp + fn) > 0 else 0
    return 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
```

## Common pitfalls

- Formatting deviations in model outputs (e.g., conversational filler, coordinate norms) are not penalized; geometric recovery is required before scoring.
- Taxonomy-aware evaluation: predicting a valid child class for a parent class query counts as correct, which may inflate scores if not accounted for.
- Model-specific prompt adaptation is necessary; using standard conversational prompts yields poor zero-shot performance for some architectures (e.g., PaliGemma requires task prefixes).

## Evidence (verbatim from paper)

> For Object Detection, Multi-Class Detection, Referring Expression Comprehension and Segmentation, we compute mIoU regardless of the output modality (HBB, OBB or mask). Crucially, evaluation is taxonomy-aware: predicting a valid child class (e.g., Church) when queried for a parent class (e.g., Building) is considered correct. We select the candidate with maximum IoU per target and report F1, Precision and Recall based on: TP: IoU ≥ 0.5 with an unclaimed ground truth. FP: IoU <0.5 or duplicate prediction for the same ground truth. FN: Unmatched ground truth object. TN: Correct rejection (no object predicted when none exists).

## Citation

```bibtex
@misc{ferrod2026groundset,
  title={GroundSet: A Cadastral-Grounded Dataset for Spatial Understanding with Vector Data},
  author={Ferrod et al. (2026)},
  year={2026},
  note={arXiv:2603.14609}
}
```

- arXiv: 2603.14609