Kaleidoscope Eval

kaleidoscope-eval

Kaleidoscope: In-language Exams for Massively Multilingual Vision Evaluation — Salazar et al. (2025) (arXiv:2504.07072, 2025)

What this evaluates

Evaluates multilingual vision-language reasoning by testing models on multiple-choice questions about images entirely in their native language. It probes cultural and linguistic authenticity, assessing how well models handle complex multimodal reasoning without relying on English translations.

Datasets

• Kaleidoscope — total 20911; splits: test (-1)

Metrics

• accuracy (primary) — range: [0, 1]
  • Proportion of valid responses where the model's selected option matches the ground truth. Computed only on questions where the model produces a valid answer in the expected format.
• Format Error Rate — range: [0, 1]
  • Proportion of total questions for which the model fails to generate a valid answer, typically due to missing the choice, selecting an invalid option, or refusing to answer.

Input / output format

Input: An image paired with a multiple-choice question (options A, B, C, D) in a target language. For closed models, the prompt includes Chain-of-Thought instructions translated into the target language. For open-weight models, the instruction is in English requesting a JSON output.

Output: For closed models: step-by-step reasoning followed by the chosen option enclosed in and tags. For open-weight models: a JSON object containing a 'choice' field with the selected option letter.

Scoring recipe

def compute_metrics(predictions, golds, model_type):
    valid_correct = 0
    valid_total = 0
    format_errors = 0
    for pred, gold in zip(predictions, golds):
        if model_type == 'closed':
            ans = extract_text(pred, '<ANSWER>', '</ANSWER>')
        else:
            ans = parse_json(pred).get('choice')
        if ans in ['A', 'B', 'C', 'D']:
            valid_total += 1
            if ans == gold: valid_correct += 1
        else:
            format_errors += 1
    accuracy = valid_correct / valid_total if valid_total > 0 else 0
    fer = format_errors / len(predictions)
    return {'accuracy': accuracy, 'format_error_rate': fer}

Common pitfalls

• Using Chain-of-Thought prompting for smaller open-weight models degrades performance and increases formatting failures compared to direct JSON output.
• Accuracy is calculated only on 'valid' responses; refusals or invalid options are excluded from the accuracy numerator but increase the Format Error Rate.
• Grouped results must report macro-average accuracy across languages to give equal weight to each language, rather than simple overall accuracy.

Evidence (verbatim from paper)

Given the multiple-choice nature of the task, we use accuracy as the primary evaluation metric. We report overall accuracy across all questions, as well as accuracy on the subset of questions where the model produces valid responses. A response is considered valid if the model successfully provides an answer in the expected format and selects a valid option (i.e., one of the letters A, B, C, D). Invalid responses typically result from missing the selected choice, selecting an invalid option, or refusal to answer. To quantify these cases, we report the Format Error Rate, which measures the proportion of questions for which the model fails to generate a valid answer.

Citation

@misc{salazar2025kaleidoscope,
  title={Kaleidoscope: In-language Exams for Massively Multilingual Vision Evaluation},
  author={Salazar et al. (2025)},
  year={2025},
  note={arXiv:2504.07072}
}

• arXiv: 2504.07072