qhjqhj00 cultural-awareness-eval
Crossroads of Continents: Automated Artifact Extraction for Cultural Adaptation with Large Multimodal Models — Mukherjee et al. (2024) (arXiv:2407.02067, 2024)
What this evaluates
Assesses the ability of large multimodal models to identify the geographical origin (country, subregion, or continent) of an image based on visual cultural cues. It probes implicit stereotypical associations and geographic bias in vision-language models.
Datasets
- Dalle Street — total 9935; splits: test (-1)
- Dollar Street — total ?; splits: test (-1)
- MaRVL — total ?; splits: test (-1)
Metrics
classification accuracy(primary) — range: percent- Percentage of correct predictions where the model's open-ended text output is mapped to a UN geoscheme region and compared against the ground truth label. Calculated at country, subregion, and continent levels, as well as union (correct at any level) and intersection (correct at all levels) accuracies.
Input / output format
Input: A single image depicting a cultural scene or artifact.
Output: Open-ended text response predicting the geographical region (country, subregion, or continent) represented in the image.
Scoring recipe
def compute_accuracy(predictions, gold_labels):
pred_regions = [map_to_un_region(p) for p in predictions]
gold_regions = gold_labels
correct = sum(1 for p, g in zip(pred_regions, gold_regions) if p == g)
return (correct / len(gold_labels)) * 100
Common pitfalls
- Models frequently trigger content policy violations instead of answering, which must be explicitly tracked as a separate category rather than treated as random errors.
- Accuracy varies significantly by geographic level; reporting only country-level accuracy underestimates model capability, as subregion and continent accuracies are substantially higher.
- Economic disparities affect performance differently across continents; lower-income quartiles show higher accuracy in Africa/Asia but lower in the Americas, requiring stratified reporting.
Evidence (verbatim from paper)
We process generated text to map it to one of the geographical sub-regions or a policy violation case and then compare it with true labels by mapping country information to geographical regions, which gives us classification accuracy as a quantitative metric for measuring success.
Citation
@misc{mukherjee2024crossroads,
title={Crossroads of Continents: Automated Artifact Extraction for Cultural Adaptation with Large Multimodal Models},
author={Mukherjee et al. (2024)},
year={2024},
note={arXiv:2407.02067}
}
- arXiv: 2407.02067