Formationeval Eval

formationeval-eval

FormationEval, an open multiple-choice benchmark for petroleum geoscience — Ermilov (2026) (arXiv:2601.02158, 2026)

What this evaluates

Evaluates large language models' domain knowledge in petroleum geoscience using a 505-question multiple-choice benchmark. It probes understanding across seven specialized subdomains, including petrophysics, reservoir engineering, and drilling, while measuring performance variance by model size, cost, and question difficulty.

Datasets

• FormationEval — total 505; splits: test (505); repo https://github.com/AlmazErmilov/FormationEval-an-Open-Benchmark-for-Oil-Gas-Geoscience-MCQ-Evaluation

Metrics

• Accuracy (primary) — range: percent
  • Percentage of correctly answered questions out of the total 505 items.

Input / output format

Input: Multiple-choice question with four options (A, B, C, D) covering one of seven petroleum geoscience domains.

Output: The model's selected option letter (e.g., 'A', 'B', 'C', or 'D').

Scoring recipe

def compute_accuracy(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if p == g)
    return (correct / len(gold)) * 100

Common pitfalls

• Residual length bias: correct answers are disproportionately the longest option, especially in hard questions (55.6% of the time).
• Medium questions have higher calculation rates (8.8%) and are more comparison-heavy, making them harder than expected relative to difficulty labels.
• Petrophysics domain consistently yields 3–5 percentage points lower accuracy due to technical well-logging concepts.

Evidence (verbatim from paper)

Accuracy spans a wide range, from 99.8% (Gemini 3 Pro Preview) to 57.6% (Llama-3.2-3b-instruct).

Citation

@misc{ermilov2026formationeval,
  title={FormationEval, an open multiple-choice benchmark for petroleum geoscience},
  author={Ermilov (2026)},
  year={2026},
  note={arXiv:2601.02158}
}

• arXiv: 2601.02158