Florence 2 Eval

florence-2-eval

Florence-2: Advancing a Unified Representation for a Variety of Vision Tasks — Bin Xiao et al. (arXiv:2311.06242, 2023)

What this evaluates

Evaluates a unified vision foundation model's zero-shot and fine-tuned capabilities across diverse computer vision tasks. It probes the model's ability to perform image captioning, visual question answering, object detection, referring expression comprehension, and semantic segmentation using a single sequence-to-sequence architecture.

Datasets

• COCO — total ?; splits: train2017 (-1), val2017 (-1), Karpathy test (-1)
• Flickr30k — total ?; splits: test (-1)
• RefCOCO/+/g — total ?; splits: val (-1), test-A (-1), test-B (-1)
• VQAv2 — total ?; splits: test-dev (-1)
• ADE20K — total ?; splits: val (-1)

Metrics

• CIDEr (primary) — range: [0, inf)
  • Consensus-based Image Description Evaluation. Computes n-gram overlap between generated captions and multiple ground-truth captions, weighted by inverse document frequency (IDF) scores.
• mAP — range: [0, 1]
  • Mean Average Precision at IoU thresholds 0.5:0.95 (APb for detection, APm for segmentation).
• Recall@1 — range: [0, 1]
  • Percentage of queries where the correct bounding box is ranked first among predictions.
• Accuracy — range: [0, 1]
  • Exact match ratio between predicted class/label and ground truth for VQA and referring expression comprehension tasks.
• mIoU — range: [0, 1]
  • Mean Intersection over Union between predicted segmentation masks and ground truth masks across all classes.

Input / output format

Input: RGB image resized to 384x384 (or 768x768 for high-res tuning) concatenated with a task-specific text prompt (e.g., '', '', '') in a sequence-to-sequence format.

Output: Text sequence generated by the decoder, representing a caption, bounding box coordinates (normalized), segmentation mask tokens, or a class label.

Scoring recipe

def score(predictions, golds, task):
    if task == 'captioning': return cider_score(predictions, golds)
    if task == 'detection': return map_score(predictions, golds, iou=0.5)
    if task == 'referring': return recall_at_1(predictions, golds)
    if task == 'vqa': return accuracy(predictions, golds)
    if task == 'segmentation': return miou_score(predictions, golds)
    return 0

Common pitfalls

• Using the standard COCO test set instead of the Karpathy split for captioning evaluation, which yields different scores.
• Comparing VQA accuracy without accounting for external OCR usage in baseline models, as some baselines use OCR tokens while Florence-2 does not.
• Confusing zero-shot evaluation (no task-specific fine-tuning) with generalist fine-tuning (single model trained on multiple public datasets).

Evidence (verbatim from paper)

For image-level tasks, Florence-2-L achieves a 135.6 CIDEr score on the COCO caption benchmark... For region-level grounding and referring expression comprehension tasks, Florence-2-L establishes a new record in zero-shot performance achieving a 5.7 improvement in Flickr30k Recall@1... Additionally, our pre-trained model attains a 35.8% mIOU in the Refcoco referring expression segmentation (RES) task

Citation

@misc{xiao2023florence2,
  title={Florence-2: Advancing a Unified Representation for a Variety of Vision Tasks},
  author={Bin Xiao et al.},
  year={2023},
  note={arXiv:2311.06242}
}

• arXiv: 2311.06242