Mirage Eval

mirage-eval

MIRAGE: A Benchmark for Multimodal Information-Seeking and Reasoning in Agricultural Expert-Guided Conversations — Dongre et al. (2025) (arXiv:2506.20100, 2025)

What this evaluates

Evaluates multimodal vision-language models on expert-level agricultural reasoning, including grounded entity identification, causal explanation quality, and dialogue management decisions (clarify vs. respond) under partial observability.

Datasets

• MIRAGE-MMST — total ?; splits: train (-1), test (-1); repo https://github.com/MIRAGE-Benchmark/MIRAGE-Benchmark
• MIRAGE-MMMT — total ?; splits: test (-1); repo https://github.com/MIRAGE-Benchmark/MIRAGE-Benchmark

Metrics

• Identification Accuracy (primary) — range: [0, 1]
  • Binary metric: 1 if the predicted entity string exactly matches any reference field (entity name, scientific name, or common names); 0 otherwise.
• Reasoning Score — range: [0, 4]
  • LLM-judge graded 0–4 scale based on the presence of key visual clues, descriptive specificity, and causal coherence in the model's justification.
• W-Sum — range: [0, 1]
  • Composite score: (2*Accuracy + Relevance + Completeness + Parsimony) / 20, aggregating four 0–4 dimension scores.
• Decision Accuracy — range: percent
  • Percentage of turns where the model's chosen action (clarify vs. respond) matches the gold annotation.

Input / output format

Input: Multimodal inputs comprising agricultural images, user queries, and dialogue history (for multi-turn tasks). Optional metadata (location, time) may be provided.

Output: Model must output a predicted biological entity, a textual justification/explanation, and a dialogue action decision (clarify or respond).

Scoring recipe

def score_mmst_id(pred, gold):
    return 1.0 if pred in [gold['name'], gold['sci_name'], gold['common']] else 0.0

def score_mmst_mg(pred, gold, judge_ensemble):
    acc = judge_ensemble.evaluate(pred, gold, rubric='factual_alignment')
    rel = judge_ensemble.evaluate(pred, gold, rubric='on_topic')
    comp = judge_ensemble.evaluate(pred, gold, rubric='covers_key_info')
    pars = judge_ensemble.evaluate(pred, gold, rubric='concise_actionable')
    return (2*acc + rel + comp + pars) / 20.0

def score_mmmt(pred_action, gold_action):
    return 1.0 if pred_action == gold_action else 0.0

Common pitfalls

• Relies on an LLM-as-a-Judge ensemble (3 models × 3 generations = 9 evaluations per sample) to ensure reliability; single-pass judge scoring will yield high variance.
• Models frequently treat provided metadata (location/time) as distracting rather than helpful, causing slight performance degradation when metadata is included.
• Significant open-world generalization gap: accuracy drops ~14 points on unseen biological entities compared to seen ones, indicating poor long-tail generalization.

Evidence (verbatim from paper)

We evaluate model performance on the MMST-ID task using two complementary metrics: 1.) Identification Accuracy which is a binary metric that measures whether the entity identified by the model matches the expert’s answer. A response is scored as correct (1) if the predicted entity string exactly matches any of the reference fields: entity name, scientific name, or common names; otherwise, it receives a score of 0. Reasoning Score, evaluates the quality of the model’s visual and linguistic justification for its prediction. It is graded on a 0–4 scale by the judges, based on the presence of key visual clues, descriptive specificity, and causal coherence.

Citation

@misc{dongre2025mirage,
  title={MIRAGE: A Benchmark for Multimodal Information-Seeking and Reasoning in Agricultural Expert-Guided Conversations},
  author={Dongre et al. (2025)},
  year={2025},
  note={arXiv:2506.20100}
}

• arXiv: 2506.20100