Pearl Eval

pearl-eval

Pearl: A Multimodal Culturally-Aware Arabic Instruction Dataset — Alwajih et al. (2025) (arXiv:2505.21979, 2025)

What this evaluates

Evaluates large vision-language models' ability to understand and generate culturally-aware Arabic content across multiple reasoning-centric question types. It probes hypothesis formation, comparative analysis, chronological reasoning, and explicit cultural grounding in both closed-form and open-ended multimodal tasks.

Datasets

• PeARL — total ?; splits: test (-1); repo https://github.com/UBC-NLP/pearl

Metrics

• relaxed-match accuracy (ACC) (primary) — range: [0, 1]
  • Binary score (1 if semantically equivalent to gold, 0 otherwise) averaged over closed-form questions. Permits synonyms, paraphrases, or minor lexical variations.
• Overall Score — range: [0.2, 5.0]
  • Weighted sum of four dimensions scored 1-5: 0.4Correctness + 0.2Coherence + 0.2Detail + 0.2Fluency.
• Cultural Awareness Score (CAS) — range: [0, 1]
  • Binary metric (0/1) indicating explicitly whether the candidate response mentions culturally-specific elements required by the reference answer.

Input / output format

Input: Multimodal prompts containing images and Arabic text questions, varying between closed-form (multiple-choice, True/False) and open-ended types (cause-and-effect, comparative analysis, scenario completion, etc.).

Output: Model-generated Arabic text responses. For closed-form, a direct answer; for open-ended, a free-form explanation or completion.

Scoring recipe

def score_closed(pred, gold):
    return 1 if is_semantically_equivalent(pred, gold) else 0

def score_open(pred, gold):
    c = judge_score(pred, gold, 'correctness')
    coh = judge_score(pred, gold, 'coherence')
    d = judge_score(pred, gold, 'detail')
    f = judge_score(pred, gold, 'fluency')
    return 0.4*c + 0.2*coh + 0.2*d + 0.2*f

def score_cas(pred, gold):
    return 1 if mentions_cultural_elements(pred, gold) else 0

Common pitfalls

• The LVLM judge's scoring may be biased by its own training data or cultural background, despite being instruction-tuned for Arabic.
• Relaxed-match accuracy relies on semantic equivalence judgments which can vary between judges or models, potentially inflating scores for paraphrased but factually incorrect answers.
• The weighted Overall Score heavily emphasizes correctness (0.4), which might overshadow coherence or fluency in open-ended evaluations.

Evidence (verbatim from paper)

For closed-form questions (e.g., multiple-choice and True/False), we utilize a relaxed-match accuracy (ACC) metric. Here, the judge assesses semantic equivalence between candidate responses and gold-standard answers, permitting synonyms, paraphrases, or minor lexical variations. Each response is assigned a binary correctness score (1 for correct, 0 for incorrect), aggregated into an overall accuracy. For open-ended questions... the judge evaluates responses using a comprehensive structured rubric capturing four critical dimensions: correctness, coherence, detail, and fluency. Each dimension is scored individually on a scale from 1 to 5, with an aggregated, weighted Overall Score calculated as follows: Overall Score = 0.4 Correctness + 0.2 Coherence + 0.2 Detail + 0.2 Fluency. We also follow Burda-Lassen et al. (2025) in employing a Cultural Awareness Score (CAS). CAS is a binary metric (0/1) indicating explicitly whether the candidate response mentions culturally-specific elements required by the reference answer...

Citation

@misc{alwajih2025pearl,
  title={Pearl: A Multimodal Culturally-Aware Arabic Instruction Dataset},
  author={Alwajih et al. (2025)},
  year={2025},
  note={arXiv:2505.21979}
}

• arXiv: 2505.21979