Gfsd Coco Eval

gfsd-coco-eval

Fast Hierarchical Learning for Few-Shot Object Detection — She et al. (2022) (arXiv:2210.05008, 2022)

What this evaluates

Evaluates a model's ability to quickly adapt an object detector to novel classes using only a few labeled examples, while preserving performance on previously learned base classes.

Datasets

• MS-COCO (G-FSD benchmark) — total ?; splits: train (-1), test (-1)

Metrics

• AP (primary) — range: percent
  • Mean Average Precision across all classes, computed as the mean of AP at IoU thresholds 0.50:0.95.
• bAP — range: percent
  • Average Precision computed only over base classes.
• nAP — range: percent
  • Average Precision computed only over novel classes.
• aAP — range: percent
  • Average Precision over the 'animal' super-class child classes.
• fAP — range: percent
  • Average Precision over the 'food' super-class child classes.

Input / output format

Input: Images with bounding box annotations for base classes and a few-shot set of novel classes.

Output: Predicted bounding boxes with class labels and confidence scores.

Scoring recipe

def compute_metric(predictions, ground_truth, class_set):
    ap_scores = []
    for cls in class_set:
        cls_preds = [p for p in predictions if p['class'] == cls]
        cls_gt = [g for g in ground_truth if g['class'] == cls]
        ap_scores.append(coco_compute_ap(cls_preds, cls_gt))
    return sum(ap_scores) / len(ap_scores)

# Run on all classes for AP, base classes for bAP, novel classes for nAP

Common pitfalls

• Evaluating on only a single random seed per shot setting can lead to high variance; the benchmark requires averaging over 10 random few-shot datasets.
• Failing to report base class performance (bAP) alongside novel class performance (nAP) hides catastrophic forgetting, which is a key evaluation criterion.
• Runtime measurements must exclude initial feature extraction time to fairly compare optimization speed.

Evidence (verbatim from paper)

We evaluate our approach on the generalized few-shot detection (G-FSD) benchmark based on the MS-COCO dataset, introduced in[33]. The benchmark uses the same class splits as in previous few-shot detection works[33, 17, 11]. We report results on 1, 2, 3, 5, and 10-shot datasets. For each K-shot setting, the benchmark samples 10 random datasets, and computes the mean scores over them in order to obtain a robust evaluation metric. For each method, we report the combined average precision scores over all classes (AP), as well as the average precision over the base (bAP) and novel (nAP) classes, separately.

Citation

@misc{she2022fasthierarchical,
  title={Fast Hierarchical Learning for Few-Shot Object Detection},
  author={She et al. (2022)},
  year={2022},
  note={arXiv:2210.05008}
}

• arXiv: 2210.05008