Codemixbench Eval

codemixbench-eval

CodeMixBench: Evaluating Code-Mixing Capabilities of LLMs Across 18 Languages — Yang et al. (2025) (arXiv:2507.18791, 2025)

What this evaluates

Evaluates large language models' ability to process and generate code-mixed text across 18 languages and 8 distinct tasks. It probes cross-lingual reasoning, traditional NLP capabilities, and few-shot learning robustness when linguistic families are mixed within a single prompt.

Datasets

• CodeMixBench — total ?; splits: test (-1); repo https://github.com/Jeromeyluck/CodeMixBench

Metrics

• accuracy (primary) — range: [0, 1]
  • Proportion of correctly predicted options, tags, or parsed final answers out of the total number of instances. Computed per task and averaged across language pairs.
• BLEU score — range: [0, 1]
  • Standard n-gram based BLEU score measuring the overlap between the model's generated translation and the reference translation.

Input / output format

Input: Code-mixed prompts or questions across 18 languages, provided in a one-shot setting with exactly one example per task included in the prompt.

Output: For multiple-choice tasks (CM-MMLU, CM-TruthfulQA): the selected option letter/text. For CM-GSM8K: chain-of-thought reasoning followed by a final answer. For LID, POS, NER, SA: JSON-formatted tags. For MT: the translated sentence.

Scoring recipe

def score(task, pred, gold):
    if task in ['CM-MMLU', 'CM-TruthfulQA', 'LID', 'POS', 'NER', 'SA']:
        return 1.0 if pred == gold else 0.0
    elif task == 'CM-GSM8K':
        final_ans = extract_regex_match(pred, r'answer is (\d+)')
        return 1.0 if final_ans == gold else 0.0
    elif task == 'MT':
        return compute_bleu(reference=gold, hypothesis=pred)
    return 0.0
metric_value = mean([score(task, p, g) for p, g in zip(predictions, golds)])

Common pitfalls

• Code-mixing between linguistically distant families (e.g., Sino-Tibetan + English) causes significant accuracy drops compared to Indo-European + English pairs, which often perform similarly to English-only baselines.
• Few-shot learning (k>1) can actually degrade performance for certain models (e.g., LLaMA2) on specific language pairs, contrary to typical expectations.
• Zero-shot mathematical reasoning (CM-GSM8K) often fails due to format non-compliance rather than reasoning errors; one-shot is required to fix output formatting.

Evidence (verbatim from paper)

For CM-MMLU and CM-TruthfulQA, we prompt models to select the correct option for multiple-choice questions. We use chain-of-thought (CoT) evaluation for CM-GSM8K task and parsed the model’s response using regular regex to obtain the final solution. We report accuracy as the evaluation metric. For above three tasks, we also provide the model performance of English-only evaluation (en only) for reference. For LID, POS, NER, and SA tasks, we prompt the models to generate the answers. Specifically, we provide the LLMs with all possible tags in the prompt and instruct models to generate in JSON format. In the MT task, we instructed models to translate code-mixed sentences. We use accuracy for LID, POS, NER, and SA tasks, and the BLEU score for MT assessment. All evaluations are under one-shot settings.

Citation

@misc{yang2025codemixbench,
  title={CodeMixBench: Evaluating Code-Mixing Capabilities of LLMs Across 18 Languages},
  author={Yang et al. (2025)},
  year={2025},
  note={arXiv:2507.18791}
}

• arXiv: 2507.18791