Phibench Eval

phibench-eval

Phi-4 Technical Report — Abdin et al. (2024) (arXiv:2412.08905, 2024)

What this evaluates

Evaluates diverse reasoning and coding capabilities using an internal benchmark designed to minimize data contamination and LLM-judge bias. It probes a model's ability to debug, extend, and explain code, as well as identify errors in mathematical proofs and generate related problems.

Datasets

• PhiBench — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: [0, 1]
  • Not explicitly defined; evaluated via LLM-as-judge with curated rubrics focusing on accuracy, logical structure, and adherence to task requirements.

Input / output format

Input: Tasks requiring model-generated outputs, including debugging code, extending incomplete code, explaining code snippets, identifying errors in mathematical proofs, and generating related math problems.

Output: Model-generated responses (code, explanations, proof corrections, problem statements) evaluated against detailed judge rubrics.

Scoring recipe

# Pseudo-code based on paper description
judge_instructions = load_curated_rubrics()
for task in PhiBench:
    response = model.generate(task.prompt)
    score = llm_as_judge(response, task.gold, judge_instructions)
    # Rubrics prioritize accuracy, logical structure, and task adherence
    # Minimize stylistic/fluency bias
    aggregate_scores(score)

Common pitfalls

• Data contamination risk from pretraining overlap, especially via rephrasing.
• LLM-as-judge scoring may prioritize style and fluency over actual reasoning accuracy if rubrics are not strictly enforced.
• Multiple-choice formats may encourage pattern matching rather than genuine reasoning.

Evidence (verbatim from paper)

For tasks requiring judgment of model-generated outputs, we addressed the common pitfalls of LLM-based scoring by carefully curating detailed judge instructions (or “judge notes”). These rubrics specify exactly how to evaluate responses, focusing on achieving accuracy, logical structure, and adherence to task requirements, while minimizing tendencies towards stylistic biases.

Citation

@misc{abdin2024phi4,
  title={Phi-4 Technical Report},
  author={Abdin et al. (2024)},
  year={2024},
  note={arXiv:2412.08905}
}

• arXiv: 2412.08905