Reef Substrate Classification Eval

reef-substrate-classification-eval

AI-driven Dispensing of Coral Reseeding Devices for Broad-scale Restoration of the Great Barrier Reef — Raine et al. (2025) (arXiv:2509.01019, 2025)

What this evaluates

Evaluates an AI model's ability to classify underwater substrates for autonomous coral reseeding deployment. It probes both fine-grained patch-level semantic segmentation (distinguishing coral, deploy, and no-deploy zones) and coarse-grained image-level decision making for real-time marine robotics.

Datasets

• Great Barrier Reef ReefScan Dataset — total ?; splits: train (-1), test (-1); repo https://github.com/sgraine/reef-guidance-system

Metrics

• Macro F1 (primary) — range: percent
  • Calculated as the unweighted average of per-class F1 scores across the three classes ('Deploy', 'No-Deploy', 'Coral'). Computed as (F1_class1 + F1_class2 + F1_class3) / 3 to ensure equal treatment of all classes despite dataset imbalance.
• Overall Accuracy — range: percent
  • Proportion of correctly classified frames or patches out of the total number of instances evaluated.

Input / output format

Input: Underwater RGB images captured by the ReefScan™ Transom Mounted Camera at depths up to 10m, processed as either full frames or spatial patches.

Output: Per-instance classification labels: 'Deploy', 'No-Deploy', or 'Coral' for patches; binary 'Deploy'/'No-Deploy' decision for whole frames.

Scoring recipe

def compute_metrics(predictions, gold):
    classes = ['Deploy', 'No-Deploy', 'Coral']
    f1s = []
    correct = 0
    for c in classes:
        tp = sum(1 for p, g in zip(predictions, gold) if p == c and g == c)
        fp = sum(1 for p, g in zip(predictions, gold) if p == c and g != c)
        fn = sum(1 for p, g in zip(predictions, gold) if p != c and g == c)
        prec = tp / (tp + fp) if (tp + fp) > 0 else 0
        rec = tp / (tp + fn) if (tp + fn) > 0 else 0
        f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
        f1s.append(f1)
        correct += tp
    macro_f1 = sum(f1s) / len(f1s)
    accuracy = correct / len(gold)
    return {'macro_f1': macro_f1, 'accuracy': accuracy}

Common pitfalls

• Using weighted F1 or standard accuracy instead of macro F1 will mask poor performance on minority classes due to severe class imbalance in reef imagery.
• Patch-level predictions must be aggregated using a tunable threshold (alpha) to make whole-frame deployment decisions; ignoring this aggregation step breaks the real-time pipeline evaluation.
• Models are evaluated on real-world reef data with varying turbidity and depth; metrics reported on synthetic or controlled lab data will not generalize to the field deployment protocol.

Evidence (verbatim from paper)

For patch classification, we report the performance of our model on three classes: ‘Deploy’, ‘No-Deploy’ and ‘Coral’, using the per-class precision, recall, F1 scores, and the overall F1 score, which we calculate as macro F1 i.e. the average of the per-class F1 scores. This ensures that all classes are treated equally, which better evaluates the performance on class imbalanced datasets. For whole image classification, we instead focus on the ‘Deploy’ precision and recall, and the overall accuracy and F1 score.

Citation

@misc{raine2025reefguidance,
  title={AI-driven Dispensing of Coral Reseeding Devices for Broad-scale Restoration of the Great Barrier Reef},
  author={Raine et al. (2025)},
  year={2025},
  note={arXiv:2509.01019}
}

• arXiv: 2509.01019