Phi 3 Academic Benchmarks Eval

phi-3-academic-benchmarks-eval

Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone — Abdin et al. (2024) (arXiv:2404.14219, 2024)

What this evaluates

Evaluates language models on reasoning, common sense, logical reasoning, knowledge retrieval, and coding capabilities across a diverse suite of standard academic benchmarks.

Datasets

• MMLU — total ?; splits: test (-1)
• HellaSwag — total ?; splits: test (-1)
• ANLI — total ?; splits: test (-1)
• GSM-8K — total ?; splits: test (-1)
• MATH — total ?; splits: test (-1)
• MedQA — total ?; splits: test (-1)
• AGIEval — total ?; splits: test (-1)
• TriviaQA — total ?; splits: test (-1)
• Arc-C — total ?; splits: test (-1)
• Arc-E — total ?; splits: test (-1)
• PIQA — total ?; splits: test (-1)
• SociQA — total ?; splits: test (-1)
• BigBench-Hard — total ?; splits: test (-1)
• WinoGrande — total ?; splits: test (-1)
• OpenBookQA — total ?; splits: test (-1)
• BoolQ — total ?; splits: test (-1)
• CommonSenseQA — total ?; splits: test (-1)
• TruthfulQA — total ?; splits: test (-1)
• HumanEval — total ?; splits: test (-1)
• MBPP — total ?; splits: test (-1)
• GPQA — total ?; splits: test (-1)
• MT Bench — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: percent
  • Percentage of correctly predicted answers out of the total number of test instances. For multiple-choice benchmarks, the model's selected option is compared to the gold label. For code benchmarks (HumanEval, MBPP), it is the pass rate of generated code against unit tests.

Input / output format

Input: A few-shot prompt template containing k examples (k varies by benchmark: 0, 2, 3, 5, 7, 8, or 10 shots) followed by the target question or task. Chain-of-thought (CoT) prompting is explicitly enabled for GSM-8K, MATH, BigBench-Hard, and GPQA.

Output: Model-generated text response. For multiple-choice benchmarks, the predicted option letter or text. For code benchmarks, a Python function. For MT Bench, a conversational response scored by an LLM judge.

Scoring recipe

def compute_accuracy(predictions, golds):
    correct = 0
    for p, g in zip(predictions, golds):
        if normalize_answer(p) == normalize_answer(g):
            correct += 1
    return (correct / len(golds)) * 100

# For MT-Bench:
# score = llm_judge_score(prediction, reference, criteria)
# return average score over 2 rounds

Common pitfalls

• Prompts use a proprietary Microsoft internal template rather than standard open-source templates, which can cause score variations compared to other evaluations.
• Chain-of-thought (CoT) is only enabled for specific benchmarks (GSM-8K, MATH, BigBench-Hard, GPQA), not universally across all tasks.
• Temperature is strictly fixed at 0 for all evaluations, which may differ from standard zero-shot or higher-temperature settings used elsewhere.

Evidence (verbatim from paper)

As is now standard, we use few-shot prompts to evaluate the models, at temperature 0. The prompts and number of shots are part of a Microsoft internal tool to evaluate language models, and in particular we did no optimization to the pipeline for the phi-3 models. All the reported numbers are produced with the exact same pipeline to ensure that the numbers are comparable.

Citation

@misc{abdin2024phi3,
  title={Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone},
  author={Abdin et al. (2024)},
  year={2024},
  note={arXiv:2404.14219}
}

• arXiv: 2404.14219