Human Scene Vlm Eval

human-scene-vlm-eval

HumanVLM: Foundation for Human-Scene Vision-Language Model — Dawei Dai et al. (2024) (arXiv:2411.03034, 2024)

What this evaluates

Evaluates a vision-language model's ability to understand and generate detailed descriptions of human-centric scenes, answer open- and closed-set questions about them, recognize facial attributes, and ground textual references to human objects in images.

Datasets

• HumanCaptionHQ — total 3950; splits: test (3950); repo https://github.com/ddw2AIGROUP2CQUPT/HumanVLM
• HumanVQA — total 18312; splits: test (18312); repo https://github.com/ddw2AIGROUP2CQUPT/HumanVLM
• FaceC — total 5000; splits: test (5000)
• CelebA — total ?; splits: test (-1)
• LFWA — total ?; splits: test (-1)
• RefCOCO — total ?; splits: testA (-1), test+ (-1)

Metrics

• semantic similarity (primary) — range: [0, 10]
  • A 0-10 score assigned by GPT-4o evaluating the semantic similarity and entity completeness between a generated caption/answer and its ground-truth reference. Two prompt variants are used (Prmt.1 and Prmt.2).
• accuracy — range: [0, 1]
  • The proportion of correctly predicted answers for closed-set VQA, facial attribute multi-label classification, and visual grounding tasks.

Input / output format

Input: An RGB image paired with a text prompt (e.g., a question for VQA, a reference phrase for grounding, or an instruction to generate a caption).

Output: Text response: a generated caption, a selected multiple-choice answer, or bounding box coordinates for visual grounding.

Scoring recipe

def score_caption(pred, gold):
    prompt = f"1.{pred} 2.{gold}"
    score = call_gpt4o(prompt, format='{"score": value}')
    return score

def score_vqa_closed(pred, gold):
    return 1.0 if pred == gold else 0.0

def score_grounding(pred_box, gold_box):
    iou = intersection_over_union(pred_box, gold_box)
    return 1.0 if iou > threshold else 0.0

Common pitfalls

• GPT-4o scoring is non-deterministic and highly sensitive to prompt phrasing; the paper reports two different prompts (Prmt.1 vs Prmt.2) yielding different scores for the same model.
• The self-constructed HumanVQA dataset uses GPT-4 to generate QA pairs, which may introduce distributional bias or overfitting to GPT-4's reasoning patterns rather than human-like question distributions.
• Accuracy on multi-label face attribute tasks (CelebA, FaceC) is reported without specifying whether it is exact match, subset accuracy, or F1 score.

Evidence (verbatim from paper)

To evaluate the ability of VLMs to interpret detailed content in human-scene images, we constructed 3,950 image-caption pairs from the HumanCaptionHQ as test data. We employed GPT4o to assess the quality of the captions generated by various models, with a focus on the completeness of entity nouns and semantic similarity. Higher scores indicate better overall performance.

Citation

@misc{dai2024humanvlm,
  title={HumanVLM: Foundation for Human-Scene Vision-Language Model},
  author={Dawei Dai et al. (2024)},
  year={2024},
  note={arXiv:2411.03034}
}

• arXiv: 2411.03034