Comprecap Eval

comprecap-eval

Benchmarking Large Vision-Language Models via Directed Scene Graph for Comprehensive Image Captioning — Fan Lu et al. (arXiv:2412.08614, 2024)

What this evaluates

Evaluates the comprehensiveness and fine-grained accuracy of detailed image captions generated by vision-language models. It probes object detection, attribute binding, directional relationship modeling, and perception of tiny objects through hierarchical scene graph alignment and dedicated VQA tasks.

Datasets

• CompreCap — total ?; splits: test (-1); repo https://github.com/LuFan31/CompreCap

Metrics

• S_object (%) — range: percent
  • Percentage score measuring the alignment and coverage of detected objects in the generated caption against the ground-truth directed scene graph.
• S_attribute — range: [0, 5]
  • Score (0-5 scale) evaluating how accurately the generated caption binds attributes to the correct objects.
• S_relation — range: [0, 5]
  • Score (0-5 scale) evaluating the accuracy of directional relationships between objects described in the caption.
• S_Cov (%) — range: percent
  • Pixel coverage percentage measuring the proportion of image area occupied by objects correctly identified and described in the caption.
• S_unified (primary) — range: [0, 100]
  • Composite comprehensive caption score (0-100) aggregating object, attribute, relation, and pixel coverage alignments to reflect overall caption quality.
• CompreQA-P ACC (%) — range: percent
  • Accuracy percentage on a fine-grained object presence VQA task, measuring the model's ability to detect and report tiny or easily ignored objects.
• CompreQA-Cap ACC (%) — range: percent
  • Accuracy percentage on a fine-grained object caption VQA task, measuring the model's ability to describe specific objects in detail.

Input / output format

Input: Image (for captioning) or Image + question (for VQA). Captioning prompt: 'Please describe the image in detail, focusing on the visible objects and the relationships among these objects.'

Output: Detailed natural language caption describing objects, attributes, and relationships, or a text/multiple-choice answer for VQA tasks.

Scoring recipe

def score_caption(image_id, pred_caption):
    gt_graph = load_scene_graph(image_id)
    pred_graph = align_caption_to_graph(pred_caption, gt_graph)
    s_obj = match_entities(gt_graph.objects, pred_graph.objects)
    s_attr = match_attributes(gt_graph.obj_attrs, pred_graph.obj_attrs)
    s_rel = match_relations(gt_graph.relations, pred_graph.relations)
    s_cov = calculate_pixel_coverage(gt_graph.objects, pred_graph.objects)
    s_unified = aggregate_scores(s_obj, s_attr, s_rel, s_cov) # 0-100
    return s_unified, s_obj, s_attr, s_rel, s_cov

def score_vqa(vqa_questions, model_answers):
    correct = sum(1 for q, a in zip(vqa_questions, model_answers) if a == q.gold)
    return correct / len(vqa_questions) * 100

Common pitfalls

• Caption length does not correlate with quality; longer outputs can still score poorly on object/attribute/relation alignment.
• Traditional n-gram metrics (BLEU, ROUGE, METEOR, CIDER) fail to capture compositional structure and show low human consistency.
• Models frequently ignore tiny objects (<5% pixel area), which heavily impacts S_Cov and S_object scores.
• Direct LLM scoring without scene graph hierarchical alignment underestimates human performance and lacks fine-grained discrimination.

Evidence (verbatim from paper)

We evaluate the detailed captions generated by these VLMs at the object level, attribute level, and relation level. ... Moreover, our $S_{\text{unified}}$ faithfully aligns with human evaluation scores compared to the result directly obtained from Llama3, shown as the last two rows of[Tab. 3].

Citation

@misc{lu2024comprecap,
  title={Benchmarking Large Vision-Language Models via Directed Scene Graph for Comprehensive Image Captioning},
  author={Fan Lu et al.},
  year={2024},
  note={arXiv:2412.08614}
}

• arXiv: 2412.08614