Litxbench Eval

litxbench-eval

LitXBench: A Benchmark for Extracting Experiments from Scientific Literature — Chong et al. (2026) (arXiv:2604.07649, 2026)

What this evaluates

Evaluates an LLM's ability to extract structured material science experimental data from scientific literature text. It specifically probes the model's capacity to link extracted measurements to material processing lineages and adhere to a predefined schema.

Datasets

• LitXAlloy — total ?; splits: test (-1)

Metrics

• F1 score (primary) — range: [0, 1]
  • Computed after using the Hungarian algorithm to find the maximum bipartite match between extracted materials and target materials. The F1 score evaluates extraction accuracy based on these optimal assignments.

Input / output format

Input: Transcribed text from scientific papers (tables in markdown, figures excluded).

Output: Code representing extracted materials according to a predefined schema (Python objects).

Scoring recipe

# 1. Use Hungarian algorithm to find max bipartite match between M_extracted and M_target
assignments = hungarian_algorithm(M_extracted, M_target, cost_function=Appendix_C_cost)
# 2. Compute precision, recall, and F1 based on optimal assignments
tp = count(matches in assignments)
fp = len(M_extracted) - tp
fn = len(M_target) - tp
precision = tp / (tp + fp) if (tp + fp) > 0 else 0
recall = tp / (tp + fn) if (tp + fn) > 0 else 0
f1 = 2 * (precision * recall) / (precision + recall) if (precision + recall) > 0 else 0

Common pitfalls

• Figures are explicitly excluded from the input, so models cannot extract data from charts or images.
• Output must be valid code adhering to a specific schema, not free-form text.
• Uncertainty is reported via 95% confidence intervals from three runs, not single-pass scores.

Evidence (verbatim from paper)

Similar to existing work (Khalighinejad et al., 2024), the Hungarian algorithm (Kuhn, 1955) is first used to find the maximum bipartite match between the set of extracted materials $M_{\text{extracted}}$ and target materials $M_{\text{target}}$. Once the optimal assignments are determined for both sets of materials, the F1 score is used to evaluate extraction accuracy, as in prior work on scientific extraction (Li et al., 2025).

Citation

@misc{chong2026litxbench,
  title={LitXBench: A Benchmark for Extracting Experiments from Scientific Literature},
  author={Chong et al. (2026)},
  year={2026},
  note={arXiv:2604.07649}
}

• arXiv: 2604.07649