Mapeval Eval

mapeval-eval

MapEval: A Map-Based Evaluation of Geo-Spatial Reasoning in Foundation Models — Dihan et al. (2024) (arXiv:2501.00316, 2024)

What this evaluates

This benchmark evaluates foundation models' geospatial reasoning capabilities across textual, visual, and API-based interaction modalities. It probes abilities such as place information retrieval, nearby point-of-interest identification, route planning, multi-step trip scheduling, and recognizing unanswerable queries.

Datasets

• MapEval — total 1000; splits: test (1000); repo https://github.com/MapEval/MapEval-API

Metrics

• accuracy (primary) — range: percent
  • Percentage of correct choices selected by the model. Calculated as (number of correct predictions / total number of questions) * 100.

Input / output format

Input: Context, question, answer format guidelines, and multiple-choice options. For MapEval-API, tool usage documentation is also provided. MapEval-Visual additionally includes map images.

Output: Multiple-choice selection (A, B, C, etc.) for Textual and Visual tasks. For API tasks, ReACT-style tool usage sequences followed by a final answer choice.

Scoring recipe

def compute_accuracy(predictions, gold_labels):
    correct = sum(1 for pred, gold in zip(predictions, gold_labels) if pred == gold)
    return (correct / len(gold_labels)) * 100

Common pitfalls

• Models frequently fail on 'Trip' planning due to difficulties in multi-step reasoning and aggregating spatio-temporal constraints.
• VLMs struggle significantly with map-specific tasks at higher zoom levels (beyond level 14) due to limited exposure to detailed map symbols and demarcations.
• API-based agents often make incorrect parameter usage errors or enter infinite request loops when no valid results are found.

Evidence (verbatim from paper)

We evaluate all tasks using the accuracy metric, defined as the percentage of correct choices selected by the model. We prompt models with the respective context, question, tool usage documentations (only for MapEval-API), answer format guidelines, and choices.

Citation

@misc{dihan2024mapeval,
  title={MapEval: A Map-Based Evaluation of Geo-Spatial Reasoning in Foundation Models},
  author={Dihan et al. (2024)},
  year={2024},
  note={arXiv:2501.00316}
}

• arXiv: 2501.00316