Main Instruction Tuning Eval

main-instruction-tuning-eval

MAIN: Mutual Alignment Is Necessary for instruction tuning — Yang et al. (2025) (arXiv:2504.12913, 2025)

What this evaluates

Evaluates the instruction-following capability, output preference quality, and general reasoning performance of fine-tuned LLMs. It probes how well models adhere to explicit constraints, generate preferred responses relative to a baseline, and solve standard academic benchmarks.

Datasets

• AlpacaEval — total 805; splits: test (805)
• IFEval — total ?; splits: test (-1)
• ARC — total ?; splits: test (-1)
• HellaSwag — total ?; splits: test (-1)
• Winogrande — total ?; splits: test (-1)
• MMLU — total ?; splits: test (-1)
• TruthfulQA — total ?; splits: test (-1)

Metrics

• AlpacaEval (primary) — range: percent
  • Percentage of pairwise comparisons where the model's output is judged better than text-davinci-003 by GPT-4. Calculated as (number of wins / total instructions) * 100.
• IFEval — range: percent
  • Accuracy across four variants: Prompt-level Strict (P-S), Instruction-level Strict (I-S), Prompt-level Loose (P-L), and Instruction-level Loose (I-L). Measures whether generated outputs satisfy explicit formatting and constraint instructions.
• OpenLLM Leaderboard accuracy — range: percent
  • Standard accuracy scores on ARC, HellaSwag, Winogrande, MMLU, and TruthfulQA, evaluated using the Language Model Evaluation Harness.

Input / output format

Input: Instruction or prompt text; for IFEval, prompts include explicit formatting/constraint instructions; for OpenLLM tasks, multiple-choice or open-ended questions.

Output: Model-generated text response.

Scoring recipe

def score_alpaca_eval(predictions, baseline_outputs):
    wins = 0
    for pred, base in zip(predictions, baseline_outputs):
        if gpt4_judge(pred, base) == 'win':
            wins += 1
    return (wins / len(predictions)) * 100

def score_ifeval(predictions, prompts):
    accuracies = []
    for pred, prompt in zip(predictions, prompts):
        accuracies.append(check_constraints(pred, prompt))
    return sum(accuracies) / len(accuracies) * 100

Common pitfalls

• AlpacaEval win rates are relative to text-davinci-003, not gold references, so scores reflect preference over a specific baseline rather than absolute quality.
• IFEval reports four distinct strict/loose variants; reporting only one metric can significantly misrepresent instruction-following capability.
• OpenLLM tasks are evaluated via the Language Model Evaluation Harness, which may apply specific prompting or few-shot templates that differ from standard zero-shot setups.

Evidence (verbatim from paper)

We assess output preference using 805 instructions from the AlpacaEval dataset. Model outputs are compared against text-davinci-003 in a pairwise setting, with GPT-4-based judgments determining win rates. Instruction-following ability is evaluated with IFEval, which reports accuracy across four metrics: Prompt-level Strict (P-S), Instruction-level Strict (I-S), Prompt-level Loose (P-L), Instruction-level Loose (I-L) ensuring a comprehensive assessment of instruction adherence.

Citation

@misc{yang2025main,
  title={MAIN: Mutual Alignment Is Necessary for instruction tuning},
  author={Yang et al. (2025)},
  year={2025},
  note={arXiv:2504.12913}
}

• arXiv: 2504.12913