Gbc Eval

gbc-eval

Graph-Based Captioning: Enhancing Visual Descriptions by Interconnecting Region Captions — Yu-Guan Hsieh et al. (2024) (arXiv:2407.06723, 2024)

What this evaluates

Evaluates the cross-modal alignment and generalization of CLIP models trained on various captioning formats. It probes zero-shot classification, bidirectional image-text retrieval, compositional reasoning, dense semantic segmentation, and fine-grained text-to-image generation control.

Datasets

• ImageNet-1k — total ?; splits: val (-1)
• Flickr30k — total ?; splits: test (-1)
• MS-COCO — total ?; splits: test (-1)
• SugarCrepe — total ?; splits: test (-1)
• ShareGPT4V-cap100k — total 15295; splits: test (15295)
• ADE20K — total ?; splits: test (-1)
• DCI — total 7805; splits: test_long (7602), test_concat (7805)

Metrics

• SugarCrepe (primary) — range: percent
  • Average accuracy across all SugarCrepe variants, measuring compositional reasoning capabilities.
• Image Retrieval Recall@K — range: [0, 1]
  • Fraction of ground-truth images retrieved within the top K ranked results from a text query.
• Text Retrieval Recall@K — range: [0, 1]
  • Fraction of ground-truth captions retrieved within the top K ranked results from an image query.
• Zero-shot Classification Accuracy — range: percent
  • Percentage of correctly predicted classes using cosine similarity between image and text embeddings without fine-tuning.
• ADE20K Segmentation mIoU — range: [0, 1]
  • Mean Intersection over Union across all semantic classes after full fine-tuning with a ViTDet UperNet head.
• DCI Retrieval Score — range: [0, 1]
  • Recall-based metric for text-to-image and image-to-text matching on filtered DCI masks and captions.

Input / output format

Input: Training: Images paired with synthetic or original captions formatted as short text, long text, region captions, GBC-captions, GBC-concat, or GBC-graph structures. Evaluation: Images with ground-truth captions/labels for retrieval, classification, segmentation, or compositional reasoning tasks.

Output: Model predictions including retrieval rankings, class probabilities, segmentation masks, or compositional reasoning answers, compared against ground-truth annotations.

Scoring recipe

def compute_metrics(predictions, gold):
    # Retrieval Recall@K
    retrieved = [pred[:K] for pred in predictions]
    recall_at_k = sum(1 for r, g in zip(retrieved, gold) if g in r) / len(gold)
    
    # Classification Accuracy
    correct = sum(1 for p, g in zip(predictions, gold) if p == g)
    accuracy = correct / len(gold)
    
    # Segmentation mIoU
    ious = [intersection_over_union(p, g) for p, g in zip(predictions, gold)]
    miou = sum(ious) / len(ious)
    
    return recall_at_k, accuracy, miou

Common pitfalls

• Filtering strategies (CLIP score quantile, topological ordering, bag-of-words fallback) are strictly applied only to the training set, not the test set.
• Dynamic batch sizing limits the number of captions/edges per batch rather than using a fixed image count, meaning reported batch sizes are upper bounds.
• DCI evaluation requires strict mask filtering (retaining only masks with bounding boxes > 224x224) and specific caption selection rules (summary vs human-annotated based on token count and order).

Evidence (verbatim from paper)

Our evaluation uses the validation set of ImageNet-1k and the test sets of Flickr30k and MS-COCO. For SugarCrepe we report the average performance across all variants. As for ShareGPT4V, we use a subset of size 15,295 from ShareGPT4V-cap100k. These images were also used for LLaVA training. When each image is paired with multiple captions, we only select one of them. The evaluation setups with ADE20K and DCI are more involved, as we explain below. Evaluation on ADE20K. We evaluate the quality of CLIP models’ image encoder for dense prediction tasks like image segmentation by performing full finetuning on ADE20k dataset. ... Evaluation on DCI. We perform text-to-image and image-to-text evaluations on DCI using either long captions or concatenated captions.

Citation

@misc{hsieh2024graphbasedcaptioning,
  title={Graph-Based Captioning: Enhancing Visual Descriptions by Interconnecting Region Captions},
  author={Yu-Guan Hsieh et al. (2024)},
  year={2024},
  note={arXiv:2407.06723}
}

• arXiv: 2407.06723