Marineeval Eval

marineeval-eval

MarineEval: Assessing the Marine Intelligence of Vision-Language Models — Wong et al. (2025) (arXiv:2512.21126, 2025)

What this evaluates

MarineEval probes the marine domain expertise and visual understanding capabilities of vision-language models. It evaluates tasks including species identification, spatial reasoning, ecological knowledge integration, and precise object localization under real-world marine conditions.

Datasets

• MarineEval — total 2000; splits: test (2000)

Metrics

• accuracy (primary) — range: [0, 1]
  • Binary classification of model outputs as correct or wrong. For Yes-No/MCQ, uses template matching or log-probability comparison over options. For localization, predicts 1 if IoU > 0.3, else 0. For open-ended/closed-form, uses an LLM judge to verify absence of missing or extraneous content against expert keypoints. Averaged across all instances.

Input / output format

Input: An image paired with a text prompt containing a question. Question formats include Yes-No, Multiple-Choice, Localization (requesting COCO bounding boxes), Closed-form, and Summarization.

Output: Text response. For MCQ/Yes-No, expected option letter/text. For Localization, COCO format (x,y,w,h). For open-ended, free-form text.

Scoring recipe

def score_instance(pred, gold, q_type):
    if q_type in ['Yes-No', 'MCQ']:
        pred_ans = extract_answer(pred)
        return 1.0 if pred_ans == gold else 0.0
    elif q_type == 'Localization':
        iou = compute_iou(pred_box, gold_box)
        return 1.0 if iou > 0.3 else 0.0
    elif q_type in ['Closed-form', 'Summarization']:
        judge_prompt = f'Check if pred misses or adds content vs gold keypoints: {gold}'
        verdict = llm_judge(judge_prompt, pred)
        return 1.0 if verdict == 'correct' else 0.0
    return 0.0

final_accuracy = mean([score_instance(p, g, t) for p, g, t in zip(preds, golds, types)])

Common pitfalls

• VLMs frequently violate strict response formats (e.g., outputting full sentences instead of option letters), requiring a log-probability fallback mechanism for fair comparison.
• Open-ended responses are evaluated via LLM judges checking for missing or extraneous content against human-summarized keypoints, which may introduce judge bias or require careful prompt calibration.
• Localization uses a fixed IoU threshold of 0.3 to binary-ize scores, which may not align with standard object detection evaluation practices.

Evidence (verbatim from paper)

To ensure objectivity, stability, and scalability, as outlined in Section[3.1], we adopt a binary judgement evaluation strategy and report the final accuracy. To clearly verify the model responses, MarineEval classifies model outputs to either correct or wrong, regardless of their format or associated capability dimensions.

Citation

@misc{wong2025marineeval,
  title={MarineEval: Assessing the Marine Intelligence of Vision-Language Models},
  author={Wong et al. (2025)},
  year={2025},
  note={arXiv:2512.21126}
}

• arXiv: 2512.21126