Uav Wildlife Tracking Eval

uav-wildlife-tracking-eval

Integrating Biological Data into Autonomous Remote Sensing Systems for In Situ Imageomics: A Case Study for Kenyan Animal Behavior Sensing with Unmanned Aerial Vehicles (UAVs) — Kline et al. (2024) (arXiv:2407.16864, 2024)

What this evaluates

This evaluation probes an autonomous UAV navigation model's ability to track wildlife by predicting flight commands that match expert pilot behavior. It measures how well the model maintains optimal camera framing and altitude for behavioral video collection.

Datasets

• KABR — total ?; splits: test (-1)

Metrics

• % of actions matching original flight (primary) — range: percent
  • Measures the percentage of predicted navigation commands that exactly match the commands issued by an expert pilot during the original flight path.
• F1 Score — range: [0, 1]
  • Harmonic mean of precision and recall for predicting navigation commands. The positive class indicates the UAV should move, while the negative class indicates hovering.

Input / output format

Input: Video frames with bounding box dimensions, UAV velocity (x,y,z), and altitude.

Output: Discrete navigation commands (e.g., 'fly left/right', 'fly forward/backward', 'change altitude', 'hover').

Scoring recipe

def compute_metrics(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if p == g)
    accuracy = correct / len(predictions)
    tp = sum(1 for p, g in zip(predictions, gold) if p == 'move' and g == 'move')
    fp = sum(1 for p, g in zip(predictions, gold) if p == 'move' and g == 'hover')
    fn = sum(1 for p, g in zip(predictions, gold) if p == 'hover' and g == 'move')
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0
    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
    return accuracy, f1

Common pitfalls

• The dataset is imbalanced in favor of 'positive' (move) commands, making accuracy an unreliable standalone metric.
• Bounding box dimensions are used as a proxy for distance, which can vary significantly with animal size and camera angle.
• Commands are discrete and must exactly match expert pilot actions; minor deviations count as errors.

Evidence (verbatim from paper)

The accuracy values reported in Table 3 simply measures the number of predicted commands that match the original flight path conducted by the expert pilot. The telemetry dataset is imbalanced in favor of 'positive' values, so the F1 Score is a better measure of performance compared to accuracy.

Citation

@misc{kline2024uavnav,
  title={Integrating Biological Data into Autonomous Remote Sensing Systems for In Situ Imageomics: A Case Study for Kenyan Animal Behavior Sensing with Unmanned Aerial Vehicles (UAVs)},
  author={Kline et al. (2024)},
  year={2024},
  note={arXiv:2407.16864}
}

• arXiv: 2407.16864