Eve Earth Intelligence Eval

eve-earth-intelligence-eval

EVE: A Domain-Specific LLM Framework for Earth Intelligence — Atrio et al. (2026) (arXiv:2604.13071, 2026)

What this evaluates

Evaluates domain-specific knowledge in Earth Observation and Earth Sciences through multiple-choice QA, hallucination detection, and open-ended QA with and without retrieval context. It also measures the preservation of general capabilities like reasoning, coding, and instruction following after domain adaptation.

Datasets

• EO and Earth Sciences Benchmark — total ?; splits: test (-1)

Metrics

• Accuracy (primary) — range: [0, 1]
  • Percentage of correctly answered single-answer MCQA questions.
• IoU — range: [0, 1]
  • Intersection over Union for multiple-answer MCQA tasks.
• F1 — range: [0, 1]
  • F1 score for the hallucination detection task.
• Judge Score — range: [0, 5]
  • Mean normalized 0–5 scale score from an LLM-as-a-judge panel (Mistral Large 3, GPT-4.1 mini, DeepSeek-V3.2, Qwen3-235B-A22B).
• Win Rate — range: [0, 1]
  • Pairwise preference win rate: WR_A = (1/N) * Σ (wins_Ai + 0.5*ties_i) / (wins_Ai + ties_i + losses_Ai).

Input / output format

Input: Question prompts, optionally accompanied by retrieved context passages. Reference answers are provided to the judge model during evaluation.

Output: Multiple-choice selections for MCQA tasks; free-text responses for open-ended QA and hallucination detection.

Scoring recipe

if task in ['MCQA_Single', 'Hallucination']:
    score = exact_match(prediction, gold) if task == 'MCQA_Single' else f1_score(prediction, gold)
elif task == 'MCQA_Multi':
    score = jaccard_score(set(prediction), set(gold))
elif task in ['Open_Ended', 'Open_Ended_Context']:
    judge_scores = [judge.score(q, pred, ref, ctx) for judge in panel]
    score = mean(normalize(judge_scores))
    # Pairwise Win Rate
    wr = sum(wins + 0.5*ties) / (wins + ties + losses) / N

Common pitfalls

• LLM-as-a-judge scores can be biased by individual model preferences; the protocol requires averaging across a 4-model panel to mitigate this.
• Win Rate calculation must correctly weight ties as 0.5 to avoid undercounting a model's performance in pairwise comparisons.
• Context availability significantly impacts open-ended QA scores; models must be evaluated both with and without retrieved context to assess true domain knowledge vs. retrieval reliance.

Evidence (verbatim from paper)

For open-ended benchmarks, we adopt the LLM-as-a-judge framework to evaluate answer correctness. Each candidate response is scored by an LLM judge conditioned on the question, reference answer, and, when applicable, retrieved context, using a 0–5 scale with predefined criteria. To improve robustness and mitigate individual model bias, we aggregate scores from a panel of judges and report the mean normalized score. Following Li et al. (2023b), we additionally conduct pairwise preference evaluation (Win Rate), where judges compare two candidate responses and select a winner or tie. The win rate of model A over model B is computed as the average preference across N evaluators: WR_A = (1/N) * Σ (wins_Ai + 0.5*ties_i) / (wins_Ai + ties_i + losses_Ai)

Citation

@misc{atrio2026eve,
  title={EVE: A Domain-Specific LLM Framework for Earth Intelligence},
  author={Atrio et al. (2026)},
  year={2026},
  note={arXiv:2604.13071}
}

• arXiv: 2604.13071