Image Transcreation Eval

image-transcreation-eval

An image speaks a thousand words, but can everyone listen? On image transcreation for cultural relevance — Khanuja et al. (2024) (arXiv:2404.01247, 2024)

What this evaluates

Evaluates multimodal models' ability to culturally adapt images (transcreation) while preserving semantics, layout, and naturalness. Probes cross-cultural visual alignment and context-aware editing capabilities for global audiences.

Datasets

• Image Transcreation Dataset — total ?; splits: concept (580), application (-1); repo https://github.com/simran-khanuja/image-transcreation

Metrics

• culture-concept (primary) — range: percent
  • Proportion of images rated as culturally representative by local human annotators for each target country.
• semantic-equivalence — range: percent
  • Proportion of images where annotators confirm the generated image belongs to the same semantic category as the original.
• spatial-layout — range: percent
  • Proportion of images where the spatial arrangement of objects is preserved after transcreation.

Input / output format

Input: Original image paired with a cultural adaptation prompt specifying the target country, concept, or application constraint (e.g., worksheet text, story text, or specific object replacement rules).

Output: Edited image intended to reflect the target culture while maintaining the original's semantic content and spatial structure.

Scoring recipe

def evaluate_transcreation(predictions, gold, annotators):
    properties = ['visual-change', 'semantic-equivalence', 'spatial-layout',
                  'culture-concept', 'naturalness', 'offensiveness',
                  'meaningful-edit', 'education-task', 'story-text', 'culture-application']
    scores = {p: 0 for p in properties}
    for img, pred in zip(predictions, gold):
        votes = [a.evaluate(img, pred) for a in annotators]
        for p in properties:
            if votes_agree_on_property(votes, p):
                scores[p] += 1
    return {p: scores[p] / len(predictions) for p in properties}

Common pitfalls

• Treating geographic tags as perfect proxies for cultural boundaries, which the authors explicitly note is a practical simplification rather than a cultural reality.
• Assuming the evaluation is fully automated; it relies on local human annotators to judge cultural relevance, naturalness, and offensiveness.
• Overlooking the strict constraints in the application split (e.g., preserving object counts for education worksheets or matching story text), which differ significantly from the looser concept split.

Evidence (verbatim from paper)

We design a two-part dataset where the first (concept) is meant to serve as a research prototype, while the second (application) is grounded in real-world applications like those in Figure [1]. Table 1: Questions asked for evaluation, the applications a model with this property would benefit (examples from Figure [1]), and the pipeline ranking for the property tested (first second third). | Does the image seem like it came from your country/ is representative of your culture? | culture-concept

Citation

@misc{khanuja2024image,
  title={An image speaks a thousand words, but can everyone listen? On image transcreation for cultural relevance},
  author={Khanuja et al. (2024)},
  year={2024},
  note={arXiv:2404.01247}
}

• arXiv: 2404.01247