# Comprecap Eval

> Evaluates the comprehensiveness and fine-grained accuracy of detailed image captions generated by vision-language models. It probes object detection, attribute binding, directional relationship modeling, and perception of tiny objects through hierarchical scene graph alignment and dedicated VQA tasks. Use when the user wants to benchmark on CompreCap, or asks about evaluating this task. Reports S_unified.

- Skill: `qhjqhj00/comprecap-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/comprecap-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/comprecap-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Productivity
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/comprecap-eval

---


# comprecap-eval

> Benchmarking Large Vision-Language Models via Directed Scene Graph for Comprehensive Image Captioning — Fan Lu et al. (arXiv:2412.08614, 2024)

## What this evaluates

Evaluates the comprehensiveness and fine-grained accuracy of detailed image captions generated by vision-language models. It probes object detection, attribute binding, directional relationship modeling, and perception of tiny objects through hierarchical scene graph alignment and dedicated VQA tasks.

## Datasets

- **CompreCap** — total ?; splits: test (-1); repo https://github.com/LuFan31/CompreCap

## Metrics

- `S_object (%)` — range: percent
  - Percentage score measuring the alignment and coverage of detected objects in the generated caption against the ground-truth directed scene graph.
- `S_attribute` — range: [0, 5]
  - Score (0-5 scale) evaluating how accurately the generated caption binds attributes to the correct objects.
- `S_relation` — range: [0, 5]
  - Score (0-5 scale) evaluating the accuracy of directional relationships between objects described in the caption.
- `S_Cov (%)` — range: percent
  - Pixel coverage percentage measuring the proportion of image area occupied by objects correctly identified and described in the caption.
- `S_unified` **(primary)** — range: [0, 100]
  - Composite comprehensive caption score (0-100) aggregating object, attribute, relation, and pixel coverage alignments to reflect overall caption quality.
- `CompreQA-P ACC (%)` — range: percent
  - Accuracy percentage on a fine-grained object presence VQA task, measuring the model's ability to detect and report tiny or easily ignored objects.
- `CompreQA-Cap ACC (%)` — range: percent
  - Accuracy percentage on a fine-grained object caption VQA task, measuring the model's ability to describe specific objects in detail.

## Input / output format

**Input**: Image (for captioning) or Image + question (for VQA). Captioning prompt: 'Please describe the image in detail, focusing on the visible objects and the relationships among these objects.'

**Output**: Detailed natural language caption describing objects, attributes, and relationships, or a text/multiple-choice answer for VQA tasks.

## Scoring recipe

```python
def score_caption(image_id, pred_caption):
    gt_graph = load_scene_graph(image_id)
    pred_graph = align_caption_to_graph(pred_caption, gt_graph)
    s_obj = match_entities(gt_graph.objects, pred_graph.objects)
    s_attr = match_attributes(gt_graph.obj_attrs, pred_graph.obj_attrs)
    s_rel = match_relations(gt_graph.relations, pred_graph.relations)
    s_cov = calculate_pixel_coverage(gt_graph.objects, pred_graph.objects)
    s_unified = aggregate_scores(s_obj, s_attr, s_rel, s_cov) # 0-100
    return s_unified, s_obj, s_attr, s_rel, s_cov

def score_vqa(vqa_questions, model_answers):
    correct = sum(1 for q, a in zip(vqa_questions, model_answers) if a == q.gold)
    return correct / len(vqa_questions) * 100
```

## Common pitfalls

- Caption length does not correlate with quality; longer outputs can still score poorly on object/attribute/relation alignment.
- Traditional n-gram metrics (BLEU, ROUGE, METEOR, CIDER) fail to capture compositional structure and show low human consistency.
- Models frequently ignore tiny objects (<5% pixel area), which heavily impacts S_Cov and S_object scores.
- Direct LLM scoring without scene graph hierarchical alignment underestimates human performance and lacks fine-grained discrimination.

## Evidence (verbatim from paper)

> We evaluate the detailed captions generated by these VLMs at the object level, attribute level, and relation level. ... Moreover, our $S_{\text{unified}}$ faithfully aligns with human evaluation scores compared to the result directly obtained from Llama3, shown as the last two rows of[Tab. 3].

## Citation

```bibtex
@misc{lu2024comprecap,
  title={Benchmarking Large Vision-Language Models via Directed Scene Graph for Comprehensive Image Captioning},
  author={Fan Lu et al.},
  year={2024},
  note={arXiv:2412.08614}
}
```

- arXiv: 2412.08614