Climateiqa Eval

climateiqa-eval

ClimateIQA: A New Dataset and Benchmark to Advance Vision-Language Models in Meteorology Anomalies Analysis — Jian Chen et al. (2024) (arXiv:2406.09838, 2024)

What this evaluates

This benchmark evaluates vision-language models on meteorological heatmap analysis, probing their ability to perform spatial localization, color semantics understanding, and anomaly detection through visual question answering. It tests four distinct capabilities: verifying statements about anomalies, enumerating affected regions, geo-indexing precise coordinates, and generating descriptive analyses.

Datasets

• ClimateIQA — total 762120; splits: train (-1), test (-1); repo https://github.com/AlexJJJChen/Climate-Zoo

Metrics

• F1 Score (primary) — range: [0, 1]
  • Harmonic mean of precision and recall for verification questions.
• Element Match Score — range: [-1, 1]
  • MS = (|x∩y| - (|x-y| + |y-x|)) / |x∪y| for non-empty sets, else 0. Measures set overlap versus symmetric difference for enumeration questions.
• Haversine Distance — range: km
  • d = 2r·arcsin(√(sin²(Δlat/2) + cos(lat_m)cos(lat_g)sin²(Δlon/2))). Measures great-circle distance between predicted and ground truth coordinates.
• BLEU / ROUGE / GPT-4o Score — range: [0, 1] | 5-point scale
  • BLEU-1/2 and ROUGE-1/2/L measure n-gram overlap. GPT-4o scores description quality on a 5-point Likert scale.

Input / output format

Input: A high-resolution meteorological heatmap image paired with a natural language question or instruction (e.g., verification, enumeration, geo-indexing, or description).

Output: A text response corresponding to the question type: a boolean/verification statement, a list of geographic locations, latitude/longitude coordinates, or a descriptive paragraph.

Scoring recipe

def score_element_match(pred_set, gold_set):
    union = pred_set | gold_set
    if len(union) == 0: return 0.0
    intersection = len(pred_set & gold_set)
    symmetric_diff = len(pred_set - gold_set) + len(gold_set - pred_set)
    return (intersection - symmetric_diff) / len(union)

def score_haversine(lat_m, lon_m, lat_g, lon_g, r=6371.0):
    dlat, dlon = math.radians(lat_m - lat_g), math.radians(lon_m - lon_g)
    a = math.sin(dlat/2)**2 + math.cos(math.radians(lat_m)) * math.cos(math.radians(lat_g)) * math.sin(dlon/2)**2
    return 2 * r * math.asin(math.sqrt(a))

Common pitfalls

• Baseline models frequently output empty sets or fail to generate valid answers, resulting in F1=0 and Element Match Score=-1, which masks true capability differences.
• Haversine distance values in the paper's tables are scaled by 10³ (reported in kilometers), which can be misread as raw meters or degrees if not checked.
• GPT-4o scoring relies on a subjective 5-point Likert scale for description quality, introducing variability compared to deterministic n-gram metrics.

Evidence (verbatim from paper)

For each question type, we adopted different evaluation metrics tailored to its specific characteristics. We detail the evaluation metrics as follows: F1 Score. For Verification Questions, we evaluate the model’s ability to judge the correctness of statements using F1 score, namely the harmonic mean of precision and recall. Element Match Score. For Enumeration Questions, we compute a match score (MS) between the ground truth ($x$) and model-generated answer ($y$). ... The score is determined by the formula: ... Haversine Distance. For Geo-indexing Questions, which involve determining precise geographical coordinates, we utilized the Haversine distance formula. ... BLEU, ROUGE and GPT-4o Scores. For Description Questions, we employ average BLEU and ROUGE and GPT-4o scores.

Citation

@misc{chen2024climateiqa,
  title={ClimateIQA: A New Dataset and Benchmark to Advance Vision-Language Models in Meteorology Anomalies Analysis},
  author={Jian Chen et al. (2024)},
  year={2024},
  note={arXiv:2406.09838}
}

• arXiv: 2406.09838