Scieval Eval

scieval-eval

SciEval: A Benchmark for Automatic Evaluation of K-12 Science Instructional Materials — Li et al. (2026) (arXiv:2604.25472, 2026)

What this evaluates

Evaluates a model's ability to automatically assess K-12 science instructional materials against pedagogical rubrics. It probes domain-aligned reasoning, long-context evidence grounding, and the capacity to generate rubric-consistent scores and justifications.

Datasets

• SciEval — total 273; splits: train (-1), val (-1), test (-1)

Metrics

• Evidence Match Rate (EMR) (primary) — range: percent
  • Proportion of predicted evidence sentences whose maximum cosine similarity to any ground-truth sentence exceeds a threshold τ=0.65. Formula: EMR = (1/|E^p|) Σ_{i=1}^{|E^p|} I[max_{e^g ∈ E^g} s(e^p_i, e^g_j) ≥ τ], where s(·,·) is sentence-embedding cosine similarity.
• Accuracy (Acc) — range: percent
  • Standard classification accuracy: proportion of correctly predicted discrete rubric scores out of total test instances.
• Quadratic Weighted Kappa (QWK) — range: percent
  • Cohen's QWK measuring inter-rater agreement between predicted and ground-truth rubric scores, penalizing larger disagreements more heavily.
• Macro-F1 (F1) — range: percent
  • Macro-averaged F1 score across all rubric score classes, balancing precision and recall equally per class.

Input / output format

Input: Extracted page-marked PDF text of the instructional material, concatenated with EQuIP rubric context and a simplified prompt template.

Output: JSON object containing a discrete integer score and a free-text string of grounded evidence sentences.

Scoring recipe

def compute_emr(pred_evidence_sentences, gold_evidence_sentences, threshold=0.65):
    if not pred_evidence_sentences:
        return 0.0
    matches = 0
    for pred_sent in pred_evidence_sentences:
        max_sim = max(cosine_similarity(pred_sent, gold_sent) for gold_sent in gold_evidence_sentences)
        if max_sim >= threshold:
            matches += 1
    return (matches / len(pred_evidence_sentences)) * 100

Common pitfalls

• EMR only captures semantic similarity and ignores page-level grounding, so models can score high while citing incorrect page numbers.
• Commercial LLMs often collapse to predicting only majority-class scores (e.g., 0 or 1), yielding high accuracy but near-zero QWK and F1.
• Long-context PDF inputs frequently cause inference failures or truncated outputs (e.g., invalid labels), which are excluded from metric calculations.

Evidence (verbatim from paper)

In AIME, models generate two outputs: a discrete score and a free-text evidence reasoning. For Score, we use Accuracy (Acc), Recall, QWK, and Macro-F1 (F1) as these traditional classification metrics that capture overall performance. Evaluating generated evidence is more challenging, as no standard metric has been established. To address this challenge, we introduce an automatic, scalable metric inspired by natural language inference (NLI), named Evidence Match Rate (EMR).

Citation

@misc{li2026scieval,
  title={SciEval: A Benchmark for Automatic Evaluation of K-12 Science Instructional Materials},
  author={Li et al. (2026)},
  year={2026},
  note={arXiv:2604.25472}
}

• arXiv: 2604.25472