Tpscalcbench Eval

tpscalcbench-eval

TPS-CalcBench: A Benchmark and Diagnostic Evaluation Framework for LLM Analytical Calculation Competence in Hypersonic Thermal Protection System Engineering — Zheng et al. (2026) (arXiv:2604.17966, 2026)

What this evaluates

Evaluates large language models' ability to perform analytical calculations in hypersonic thermal protection system engineering using closed-form formulas and thermodynamic relations, without relying on external simulation tools.

Datasets

• TPS-CalcBench — total 420; splits: train (-1), test (-1)

Metrics

• relative_error (primary) — range: [0, 1]
  • Calculated as |prediction - ground_truth| / ground_truth for each target quantity. Measures outcome correctness.
• unit_verification — range: [0, 1]
  • Binary check verifying whether the model's predicted answer matches the expected physical unit specified in the target metadata.
• process_trustworthiness — range: [0, 1]
  • Scored via an 8-dimension rubric evaluated by calibrated LLM judges to assess reasoning steps and detect 'right answer, wrong reasoning' failures.

Input / output format

Input: Structured JSON containing problem statement (question), given parameters (name, value, unit, description), target quantities (name, expected unit, scoring weight, description), difficulty level, task type, and domain tags.

Output: Numerical result for the target quantity, optionally accompanied by derivation steps or reasoning trace for process evaluation.

Scoring recipe

def score(prediction, gold, metadata):
    pred_val = extract_numeric(prediction)
    gold_val = gold['value']
    rel_err = abs(pred_val - gold_val) / gold_val
    unit_ok = 1.0 if prediction.unit == metadata['expected_unit'] else 0.0
    process_score = llm_judge_rubric(prediction.reasoning_trace, metadata)
    return rel_err, unit_ok, process_score

Common pitfalls

• Data leakage from textbook chapter patterns if not strictly source-grouped stratified.
• Including simulation-dependent problems that violate the analytical scope boundary.
• Failing to detect 'right answer, wrong reasoning' without process-level rubric scoring.

Evidence (verbatim from paper)

It establishes a dual-axis evaluation framework—outcome correctness (via relative error and unit verification) and process trustworthiness (via an 8-dimension rubric scored by calibrated LLM judges)—to detect 'right answer, wrong reasoning' failures.

Citation

@misc{zheng2026tpscalcbench,
  title={TPS-CalcBench: A Benchmark and Diagnostic Evaluation Framework for LLM Analytical Calculation Competence in Hypersonic Thermal Protection System Engineering},
  author={Zheng et al. (2026)},
  year={2026},
  note={arXiv:2604.17966}
}

• arXiv: 2604.17966