Cl Ifeval Eval

cl-ifeval-eval

Multi-lingual Functional Evaluation for Large Language Models — Ojewale et al. (2025) (arXiv:2506.20793, 2025)

What this evaluates

Evaluates large language models' ability to follow complex, variable-driven instructions across multiple languages. It measures strict compliance with prompt constraints to reveal cross-lingual robustness disparities. The benchmark highlights how functional tasks expose performance gaps that static benchmarks often miss.

Datasets

• CL-IFEval — total ?; splits: test (-1)

Metrics

• Strict Prompt Accuracy (primary) — range: percent
  • Percentage of instances where the model's output exactly matches all required constraints and formatting specified in the prompt.

Input / output format

Input: Instruction prompts translated into English, French, Spanish, Hindi, Arabic, or Yoruba, containing dynamic variables and functional constraints.

Output: Model-generated text responses that must satisfy all specified constraints.

Scoring recipe

For each instance, parse the model output and check if it satisfies all constraints defined in the prompt.
Mark as correct (1) if all constraints are met, else incorrect (0).
Average across instances to get Strict Prompt Accuracy.

Common pitfalls

• Strict accuracy requires exact compliance with all constraints; partial compliance is scored as 0.
• Performance varies significantly across languages, with low-resource languages like Yoruba showing drastically lower scores than high-resource ones.
• Model rankings on functional benchmarks differ from static benchmarks, so cross-benchmark comparisons can be misleading.

Evidence (verbatim from paper)

Figure 3: Cross-Lingual IFEval Strict Prompt Accuracy.

Citation

@misc{ojewale2025multilingual,
  title={Multi-lingual Functional Evaluation for Large Language Models},
  author={Ojewale et al. (2025)},
  year={2025},
  note={arXiv:2506.20793}
}

• arXiv: 2506.20793