Mmlu Bbh Gsm8k Eval

mmlu-bbh-gsm8k-eval

Fine-Tuning on Noisy Instructions: Effects on Generalization and Performance — Alajrami et al. (2025) (arXiv:2510.03528, 2025)

What this evaluates

Evaluates how fine-tuning large language models on syntactically or semantically perturbed instructions impacts downstream generalization across factual knowledge, complex reasoning, and mathematical problem-solving. It also measures potential side effects on model toxicity and truthfulness under varying noise levels during both training and evaluation.

Datasets

• MMLU — total ?; splits: test (-1)
• BBH — total ?; splits: test (-1)
• GSM8K — total 8500; splits: test (-1)
• ToxiGen — total ?; splits: test (-1)
• TruthfulQA — total ?; splits: test (-1)

Metrics

• average test accuracy (primary) — range: percent
  • Average test accuracy across 57 subjects, computed as the proportion of correctly answered multiple-choice questions in 0-shot and 5-shot settings.
• average exact match — range: percent
  • Average exact match across 23 sub-tasks, comparing model output to ground truth answers using direct prompting or chain-of-thought with 3 in-context examples.
• exact match — range: percent
  • Exact match between model-generated final answer and ground truth for grade-school math word problems, evaluated with direct prompting or chain-of-thought using 8 in-context examples.
• toxicity rate — range: percent
  • Percentage of generated outputs classified as toxic or hate speech by a fine-tuned RoBERTa classifier across demographic prompts.
• truthfulness score — range: other
  • Truthfulness and informativeness scores measured by off-the-shelf Llama-2 (7B) judge models, assessing the model's ability to avoid known falsehoods while remaining informative.

Input / output format

Input: Multiple-choice questions (MMLU), reasoning/math word problems (BBH, GSM8K), demographic prompts (ToxiGen), and misconception prompts (TruthfulQA), provided with 0-shot, 5-shot, or few-shot chain-of-thought examples depending on the benchmark.

Output: Text responses generated by the model, which are either directly compared to ground truth for exact match/accuracy, or passed to specialized judge models/classifiers for toxicity and truthfulness scoring.

Scoring recipe

def compute_metrics(predictions, golds, metric_type):
    if metric_type in ['accuracy', 'exact_match']:
        correct = sum(1 for p, g in zip(predictions, golds) if normalize(p) == normalize(g))
        return correct / len(golds) * 100
    elif metric_type == 'toxicity':
        toxic_count = sum(1 for p in predictions if classifier.predict(p) == 'toxic')
        return toxic_count / len(predictions) * 100
    elif metric_type == 'truthfulness':
        scores = [judge_model.score(p) for p in predictions]
        return sum(scores) / len(scores)

Common pitfalls

• Confusing the perturbation rate used during fine-tuning with the perturbation rate used during evaluation; both are independently varied at 0%, 25%, 50%, 75%, and 100%.
• Assuming identical few-shot prompting across benchmarks; BBH uses 3 in-context examples while GSM8K uses 8, and both support direct vs. CoT prompting.
• Ignoring reported standard deviations over three runs when claiming performance improvements, as results vary across random seeds.

Evidence (verbatim from paper)

We assess downstream performance using: Massive Multitask Language Understanding (MMLU; Hendrycks et al. [2021]): MMLU evaluates a model’s factual knowledge and reasoning across 57 subjects, ranging from elementary to professional-level difficulty, using multiple-choice questions. We follow the original MMLU setup, evaluating in 0-shot and 5-shot settings, and report average test accuracy. Big-Bench Hard (BBH; Suzgun et al. [2022]): A challenging subset of 23 tasks from the original BIG-Bench (Srivastava et al., [2023]), aimed at evaluating advanced reasoning in language models. We assess both direct prompting and chain-of-thought (CoT) Wei et al. ([2022b]), using official prompts with three in-context examples, and report average exact match across sub-tasks. Grade School Math (GSM8K; Cobbe et al. [2021]): A benchmark of 8.5K grade school-level word problems for testing multi-step mathematical reasoning in language models. We evaluate with direct prompting and CoT using eight in-context few-shot examples, and we report the exact match.

Citation

@misc{alajrami2025finetuning,
  title={Fine-Tuning on Noisy Instructions: Effects on Generalization and Performance},
  author={Alajrami et al. (2025)},
  year={2025},
  note={arXiv:2510.03528}
}

• arXiv: 2510.03528