Camera Trap AI Eval

camera-trap-ai-eval

Choosing an Appropriate Platform and Workflow for Processing Camera Trap Data using Artificial Intelligence — Vélez et al. (2022) (arXiv:2202.02283, 2022)

What this evaluates

Evaluates AI-powered platforms for processing camera trap images, measuring their ability to detect animals and classify species against ground truth labels.

Datasets

• Colombian rainforest camera trap images — total 112247; splits: test (112247)

Metrics

• precision — range: [0, 1]
  • Precision = TP / (TP + FP). Measures the proportion of predicted positive instances that are actually correct.
• recall — range: [0, 1]
  • Recall = TP / (TP + FN). Measures the proportion of actual positive instances that are correctly identified.
• F1 score (primary) — range: [0, 1]
  • F1 = 2 * (Precision * Recall) / (Precision + Recall). Harmonic mean of precision and recall, used as the headline composite metric.

Input / output format

Input: Raw camera trap images.

Output: Predicted species labels or object categories (e.g., 'Animal', 'Blank') with associated confidence scores.

Scoring recipe

def evaluate(predictions, gold, ct):
    tp = fp = fn = 0
    for pred, true_label in zip(predictions, gold):
        if pred.confidence >= ct:
            if pred.label == true_label:
                tp += 1
            else:
                fp += 1
        else:
            fn += 1
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0
    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
    return precision, recall, f1

Common pitfalls

• Species classifiers often exhibit high precision but very low recall (<54%), meaning they miss many actual animals despite being accurate when they do predict.
• Performance is highly sensitive to the confidence threshold; lowering it increases recall but decreases precision, requiring careful trade-off selection.
• Platforms are best used in semi-automated workflows requiring expert review of low-confidence or 'Blank' predictions rather than fully automated deployment.

Evidence (verbatim from paper)

WI and MLWIC2 had high precision values for some species (at a CT = 0.65), suggesting that when computer vision predicted a species label, it was usually correct. However, all species had low recall values (less than 54%, at a CT = 0.65), indicating that these platforms missed many of the animals present in the images. MD, which identifies broader categories of objects, had a precision of 98% at a 93% recall (at a CT = 0.65) for the "Animal" class, and consequently also had a high F1 score.

Citation

@misc{velez2022choosing,
  title={Choosing an Appropriate Platform and Workflow for Processing Camera Trap Data using Artificial Intelligence},
  author={Vélez et al. (2022)},
  year={2022},
  note={arXiv:2202.02283}
}

• arXiv: 2202.02283