Nautilus Eval

nautilus-eval

NAUTILUS: A Large Multimodal Model for Underwater Scene Understanding — Wei Xu et al. (2025) (arXiv:2510.27481, 2025)

What this evaluates

Evaluates large multimodal models on underwater scene understanding across eight tasks, including coarse/fine classification, image/region captioning, grounding, detection, VQA, and object counting. It probes the model's robustness to severe underwater image degradation (light scattering, absorption, color casts) and its ability to generalize to unseen underwater domains.

Datasets

• NautData — total 1450000; splits: train (-1), test (-1); repo https://github.com/H-EmbodVis/NAUTILUS
• IOCfish5k — total 5000; splits: train (-1), test (-1)
• MarineInst20M — total 20000000; splits: human-annotated (-1)

Metrics

• accuracy (primary) — range: percent
  • Percentage of correctly predicted labels or counts. Computed as the number of exact matches between predictions and ground truth divided by the total number of instances.
• METEOR — range: [0, 1]
  • Standard METEOR metric for evaluating text generation quality, measuring unigram precision, recall, and a penalty for fragmentation.
• mIoU — range: [0, 1]
  • Mean Intersection over Union across all classes or instances, measuring the overlap between predicted and ground-truth bounding boxes or regions.
• PR@0.5 — range: [0, 1]
  • Precision-Recall curve area or precision at a fixed recall threshold of 0.5, used for grounding and detection evaluation.
• mAP@0.5 — range: [0, 1]
  • Mean Average Precision at an IoU threshold of 0.5, standard for object detection benchmarks.
• MAE — range: other
  • Mean Absolute Error, calculated as the average of absolute differences between predicted and true counts.
• RMSE — range: other
  • Root Mean Squared Error, calculated as the square root of the average squared differences between predicted and true counts.

Input / output format

Input: Underwater image paired with a natural language instruction specifying the task (e.g., classification prompt, captioning request, grounding query, detection instruction, VQA question, or counting question).

Output: Text response containing the predicted class label, generated caption, bounding box coordinates, integer count, or VQA answer.

Scoring recipe

def compute_metrics(predictions, golds, task):
    if task == 'classification':
        return sum(p == g for p, g in zip(predictions, golds)) / len(golds)
    elif task in ['caption', 'vqa']:
        return compute_METEOR(predictions, golds)
    elif task == 'grounding':
        pred_boxes = parse_boxes(predictions)
        return compute_mIoU(pred_boxes, golds), compute_PR_at_threshold(pred_boxes, golds, 0.5)
    elif task == 'detection':
        pred_boxes = parse_boxes(predictions)
        return compute_mAP(pred_boxes, golds), compute_mAP_at_threshold(pred_boxes, golds, 0.5)
    elif task == 'counting':
        pred_counts = [int(p) for p in predictions]
        mae = mean(abs(p - g) for p, g in zip(pred_counts, golds))
        rmse = sqrt(mean((p - g)**2 for p, g in zip(pred_counts, golds)))
        acc = sum(p == g for p, g in zip(pred_counts, golds)) / len(golds)
        return mae, rmse, acc

Common pitfalls

• Standard LMMs fail significantly on underwater data due to severe degradation (light scattering, absorption, color casts), requiring domain-specific adaptation rather than direct zero-shot application.
• The counting task uses MAE and RMSE as primary metrics due to its regression nature, not just accuracy, which can mask performance differences.
• Zero-shot evaluations on MarineInst20M use only the human-annotated subsets (Flickr, Shutterstock, Gettyimages), not the full 20M dataset.
• Ablation studies train on only 1/3 of NautData's training set, which may not reflect full-scale performance or generalization.

Evidence (verbatim from paper)

Experiments are primarily conducted on the NautData. MarineInst20M is a recently impressive underwater vision-language dataset containing high-quality image-caption pairs. Among this dataset, we conduct zero-shot evaluations on its human-annotated part involving the Flickr, Shutterstock, and Gettyimages subsets. IOCfish5k is the unique underwater object-counting dataset among the collected datasets. We evaluate the counting performance on its test set. Table 2: The comparison of our Nautilusand renowned LMMs on the NautData test set. ... acc↑ METEOR↑ mIoU↑ PR@0.5↑ mAP↑ mAP@0.5↑

Citation

@misc{xu2025nautilus,
  title={NAUTILUS: A Large Multimodal Model for Underwater Scene Understanding},
  author={Wei Xu et al. (2025)},
  year={2025},
  note={arXiv:2510.27481}
}

• arXiv: 2510.27481