Trans Env Eval

trans-env-eval

Trans-EnV: A Framework for Evaluating the Linguistic Robustness of LLMs Against English Varieties — Lee et al. (2025) (arXiv:2505.20875, 2025)

What this evaluates

Evaluates the linguistic robustness of LLMs by measuring performance degradation when standard English prompts are transformed into 38 regional dialects and ESL varieties. It probes whether models maintain accuracy and instruction-following capabilities across non-standard linguistic variations.

Datasets

• MMLU — total ?; splits: (unstated)
• ARC — total ?; splits: (unstated)
• TruthfulQA — total ?; splits: (unstated)
• GSM8K — total ?; splits: (unstated)
• HellaSwag — total ?; splits: (unstated)
• WinoGrande — total ?; splits: (unstated)
• IFEval — total ?; splits: (unstated)
• AlpacaFarm — total ?; splits: (unstated)
• MT-Bench — total ?; splits: (unstated)

Metrics

• accuracy (primary) — range: percent
  • Percentage of correctly answered instances. For closed-form QA, it is the proportion of predictions matching the ground truth. For IFEval, it is the accuracy of instruction adherence.
• win_rate — range: percent
  • Proportion of times the evaluated model's output is preferred over GPT-4's output in pairwise comparisons.
• llm_judge_score — range: [0, 10]
  • Score from 0 to 10 assigned by an LLM-as-a-judge indicating how well the model’s output aligns with the given instruction.

Input / output format

Input: Prompt or question text transformed into a specific English variety (18 regional dialects or 20 ESL forms) using the Trans-EnV framework.

Output: Model-generated answer or response text.

Scoring recipe

def compute_accuracy(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if is_match(p, g))
    return (correct / len(predictions)) * 100

# For open-ended tasks:
# IFEval: accuracy of instruction adherence
# AlpacaFarm: win_rate vs GPT-4
# MT-Bench: llm_judge_score (0-10) from GPT-4.1

Common pitfalls

• Performance degradation is heavily influenced by pretraining data exposure for each variety, so drops may reflect exposure bias rather than pure linguistic robustness.
• CEFR level A varieties sometimes yield lower scores than B due to higher syntactic deviation from SAE, contradicting the assumption that simpler English is easier for models.
• Open-ended evaluation relies on LLM-as-a-judge, making results sensitive to the specific judge model version and prompt used.

Evidence (verbatim from paper)

For IFEval, the metric is the accuracy of instruction adherence. For AlpacaFarm, we report the win rate when comparing model outputs against GPT-4. For MT-Bench, the evaluation LLM provides a score from 0 to 10 indicating how well the model’s output aligns with the given instruction.

Citation

@misc{lee2025transenv,
  title={Trans-EnV: A Framework for Evaluating the Linguistic Robustness of LLMs Against English Varieties},
  author={Lee et al. (2025)},
  year={2025},
  note={arXiv:2505.20875}
}

• arXiv: 2505.20875