Aicabench Eval

aicabench-eval

AICA-Bench: Holistically Examining the Capabilities of VLMs in Affective Image Content Analysis — Dong She et al. (arXiv:2604.05900, 2026)

What this evaluates

Evaluates Vision-Language Models on affective image content analysis across three dimensions: Emotion Understanding (identifying emotions in images), Emotion Reasoning (inferring emotional causes/context), and Emotion-Guided Content Generation (producing text guided by emotional intent). It probes models' ability to perceive, reason about, and generate content based on visual emotional cues, including sensitivity to abstract art and reliance on facial shortcuts.

Datasets

• AICA-Bench — total ?; splits: test (-1)

Metrics

• accuracy / F1 (%) (primary) — range: [0, 100]
  • Percentage of correctly predicted labels or generated outputs per task dimension (EU, ER, EGCG). Scores are averaged across sub-tasks to produce EU Avg., ER Avg., EG Avg., and an Overall Avg. The paper also reports F1 scores for specific ablation studies.

Input / output format

Input: Single image paired with a task-specific prompt (Emotion Understanding, Emotion Reasoning, or Emotion-Guided Content Generation).

Output: Text response generated by the VLM.

Scoring recipe

# Pseudo-code based on table structure and paper description
preds = model.generate(image, prompt)
gold = get_ground_truth(task_type)
if task_type in ["EU", "ER"]:
    score = exact_match_or_llm_judge(preds, gold)
elif task_type == "EGCG":
    score = generation_quality_score(preds, gold)
EU_Avg = mean(EU_Basic, EU_CoT)
Overall_Avg = mean(EU_Avg, ER_Avg, EG_Avg)

Common pitfalls

• Models heavily rely on facial expressions as a visual shortcut; masking faces causes an ~11% F1 drop, indicating poor holistic context integration.
• Performance degrades significantly on abstract art compared to realistic photos, revealing a lack of affective transfer capability.
• Simply scaling model parameters (e.g., 8B to 16B) yields negligible gains, showing that fine-grained visual encoding, not size, is the bottleneck.

Evidence (verbatim from paper)

We benchmark a total of 23 VLMs, comprising 7 closed-source commercial models via APIs and 16 open-source models ranging from 2B to 16B parameters. Table 3: Main results on the AICA-Bench benchmark. EU: Emotion Understanding, ER: Emotion Reasoning, EGCG: Emotion-guided Content Generation. ... when high-level facial cues are occluded, the model’s performance suffers a sharp decline, with the F1 score dropping by 11.1%.

Citation

@misc{she2026aicabench,
  title={AICA-Bench: Holistically Examining the Capabilities of VLMs in Affective Image Content Analysis},
  author={Dong She et al.},
  year={2026},
  note={arXiv:2604.05900}
}

• arXiv: 2604.05900