qhjqhj00 gpt4-based-exact-match
Orca-Math: Unlocking the potential of SLMs in Grade School Math — Mitra et al. (2024) (arXiv:2402.14830, 2024)
What this evaluates
Evaluates whether a language model's final numerical answer to a grade-school math word problem matches the ground truth. It uses an external LLM to extract the final answer from the model's generated solution and compares it against the gold answer.
Datasets
- GSM8K — total ?; splits: test (-1)
Metrics
GPT4-based-Exact-Match(primary) — range: [0, 1]- An LLM (GPT-4) is prompted to extract the final numerical answer from the model's generated solution and compare it to the gold answer. If they match, the prediction is marked Correct; otherwise, Incorrect.
Input / output format
Input: Model-generated solution text, problem question, and gold solution text.
Output: A structured response containing 'Error Analysis' (one sentence comparing answers) and 'Final Verdict' (Correct or Incorrect).
Scoring recipe
def compute_metric(predictions, gold_answers):
correct_count = 0
for pred, gold in zip(predictions, gold_answers):
# GPT-4 extracts final answer from pred and compares to gold
verdict = call_gpt4_extract_and_compare(pred, gold)
if verdict == "Correct":
correct_count += 1
return correct_count / len(predictions)
Common pitfalls
- Relies on GPT-4 for answer extraction, which may introduce latency/cost and potential extraction errors compared to regex-based exact match.
- Only checks the final numerical answer, ignoring intermediate reasoning steps or alternative valid solution paths.
Evidence (verbatim from paper)
We use exact match as the metric. Given a model generated answer, we prompt GPT4 to extract the final short answer and match it with the gold short answer. We will refer to this metric as, GPT4-based-Exact-Match.
Citation
@misc{mitra2024orcamath,
title={Orca-Math: Unlocking the potential of SLMs in Grade School Math},
author={Mitra et al. (2024)},
year={2024},
note={arXiv:2402.14830}
}
- arXiv: 2402.14830