Coco Unifs Eval

coco-unifs-eval

UniFS: Universal Few-shot Instance Perception with Point Representations — Sheng Jin et al. (arXiv:2404.19401, 2024)

What this evaluates

Evaluates a model's ability to perform universal few-shot instance perception across object detection, instance segmentation, pose estimation, and object counting. It probes task-agnostic generalization and robustness in extremely low-shot (1-shot and 5-shot) scenarios, including unseen-task generalization for counting.

Datasets

• COCO-UniFS — total ?; splits: train (-1), val (-1); repo https://github.com/jin-s13/UniFS
• PASCAL-5i — total ?; splits: train (-1), test (-1)

Metrics

• Det. AP (primary) — range: percent
  • Average Precision for object detection, averaged over IoU thresholds 0.50 to 0.95.
• Seg. AP — range: percent
  • Average Precision for instance segmentation masks.
• Kpt. AP — range: percent
  • Average Precision for pose estimation keypoints.
• Cnt. MSE — range: other
  • Mean Squared Error between predicted and ground-truth object counts.
• AP_{50} — range: percent
  • Average Precision at a fixed IoU threshold of 0.50.

Input / output format

Input: RGB images with support images containing K-shot annotations (bounding boxes, masks, or keypoints) for novel categories, plus query images for evaluation.

Output: Point representations corresponding to object centers, segmentation masks, keypoints, or count estimates, depending on the task.

Scoring recipe

def compute_metrics(predictions, gold, task):
    if task == 'counting':
        return mean_squared_error(predictions.counts, gold.counts)
    elif task in ['detection', 'segmentation', 'pose']:
        matches = match_predictions_to_ground_truth(predictions, gold, iou_thresholds=[0.50, 0.55, ..., 0.95])
        precisions, recalls = compute_precision_recall_curves(matches)
        return interpolate_ap(precisions, recalls)
    return None

Common pitfalls

• Baseline models are trained on the counting task during base training, while UniFS is evaluated on it as an unseen task; comparing them directly without noting this distinction misrepresents generalization capability.
• Results are averaged over 10 random seeds due to stochastic task selection during training; reporting single-run results introduces high variance.
• The few-shot fine-tuning uses a fixed learning rate of 0.01 and K-shots per class; failing to match this exact fine-tuning protocol leads to unfair comparisons.

Evidence (verbatim from paper)

We report AP for object detection (Det.), instance segmentation (Seg.), and pose estimation (Kpt.), and report MSE for object counting (Cnt.). The results are averaged over all 10 seeds for fair comparisons.

Citation

@misc{jin2024unifs,
  title={UniFS: Universal Few-shot Instance Perception with Point Representations},
  author={Sheng Jin et al.},
  year={2024},
  note={arXiv:2404.19401}
}

• arXiv: 2404.19401