Infinity Instruct Eval

infinity-instruct-eval

Infinity Instruct: Scaling Instruction Selection and Synthesis to Enhance Language Models — Li et al. (2025) (arXiv:2506.11116, 2025)

What this evaluates

Evaluates the conversational and foundational capabilities of LLMs fine-tuned on instruction datasets, comparing them against proprietary and open-source baselines across multiple standard benchmarks.

Datasets

• AlpacaEval 2.0 — total ?; splits: test (-1)
• Arena-Hard — total ?; splits: test (-1)
• MT-Bench — total ?; splits: test (-1)
• MATH — total ?; splits: test (-1)
• GSM-8K — total ?; splits: test (-1)
• HumanEval — total ?; splits: test (-1)
• MBPP — total ?; splits: test (-1)
• MMLU — total ?; splits: test (-1)
• LUC-EVAL — total ?; splits: test (-1)

Metrics

• AlpacaEval 2.0 — range: percent
  • Win rate percentage against a reference model (typically GPT-4-turbo) using LLM-as-judge.
• Arena-Hard — range: percent
  • Average score from pairwise comparisons judged by LLMs, normalized to 0-100.
• MT-Bench — range: percent
  • Average score across multi-turn conversations rated by LLMs, mapped to 0-100 for reporting.
• MATH — range: percent
  • Exact-match accuracy on mathematical reasoning problems.
• GSM-8K — range: percent
  • Exact-match accuracy on grade-school math word problems.
• HumanEval — range: percent
  • Pass@1 accuracy based on unit test execution.
• MBPP — range: percent
  • Pass@1 accuracy based on unit test execution for code generation.
• MMLU — range: percent
  • Exact-match accuracy on multiple-choice questions across 57 subjects.
• LUC-EVAL — range: percent
  • Accuracy on a specialized evaluation suite for foundational capabilities.
• Overall* (primary) — range: percent
  • Average of normalized scores across all listed benchmarks. MT-Bench scores are explicitly mapped to 0-100 before averaging.

Input / output format

Input: Instruction prompts or conversational turns from evaluation benchmarks.

Output: Model-generated text responses.

Scoring recipe

def compute_metrics(predictions, golds, benchmark):
    if benchmark in ['AlpacaEval 2.0', 'Arena-Hard', 'MT-Bench']:
        score = llm_judge_score(predictions, golds)
    else:
        score = exact_match_or_execution_accuracy(predictions, golds)
    return score

def compute_overall(predictions, golds, benchmarks):
    normalized_scores = []
    for b in benchmarks:
        s = compute_metrics(predictions, golds, b)
        if b == 'MT-Bench':
            s = s * 10  # Map 0-10 to 0-100
        normalized_scores.append(s)
    return sum(normalized_scores) / len(normalized_scores)

Common pitfalls

• MT-Bench scores are originally on a 0-10 scale but must be multiplied by 10 to match the 0-100 scale of other benchmarks in the overall average.
• AlpacaEval 2.0 reports win rates against a specific reference model (usually GPT-4-turbo), so scores are not absolute but relative.
• Arena-Hard and MT-Bench rely on LLM-as-judge evaluations, which can introduce bias depending on the judge model and prompt.

Evidence (verbatim from paper)

To calculate the overall average score, we mapped MT-Bench scores to 0-100.

Citation

@misc{li2025infinityinstruct,
  title={Infinity Instruct: Scaling Instruction Selection and Synthesis to Enhance Language Models},
  author={Li et al. (2025)},
  year={2025},
  note={arXiv:2506.11116}
}

• arXiv: 2506.11116