ifc-bench-v2-eval
BIM Information Extraction Through LLM-based Adaptive Exploration — Hellin et al. (2026) (arXiv:2605.01698, 2026)
What this evaluates
Tests an LLM's ability to extract, compute, and reason over heterogeneous Building Information Modeling (BIM) data (IFC files) using adaptive code execution or static baselines. It probes robustness to data heterogeneity, documentation retrieval, and tool augmentation.
Datasets
Metrics
aggregate accuracy (primary) — range: [0, 1]
- A non-abstained answer is rated correct only when all four substantive criteria (faithfulness, completeness, transparency, relevance) are simultaneously satisfied. Formula: accuracy = |{i: c1,i ∧ c2,i ∧ c3,i ∧ c4,i}| / N.
Input / output format
Input: Natural-language question, reference IFC model path, and optionally pre-fetched documentation or available tools.
Output: A final natural-language answer or an explicit abstention message.
Scoring recipe
def compute_aggregate_accuracy(predictions, gold, criteria):
correct = 0
for i in range(len(predictions)):
if not predictions[i].abstained:
if all(criteria[i][c] for c in [1, 2, 3, 4]):
correct += 1
return correct / len(predictions)
Common pitfalls
- Abstention is distinct from system errors (crashes, timeouts), which are excluded from evaluation.
- Faithfulness thresholds are category-dependent: Category 1 requires grounding in BIM data only, while Category 4 permits stated assumptions alongside data evidence.
- The metric is deliberately conservative; a single criterion failure across the four substantive criteria marks the entire answer incorrect.
Evidence (verbatim from paper)
The primary metric reported throughout this study is aggregate accuracy: a non-abstained answer is rated correct only when all four substantive criteria (1–4) are simultaneously satisfied: accuracy = |{i: c1,i ∧ c2,i ∧ c3,i ∧ c4,i}| / N. This metric is deliberately conservative: a single criterion failure marks the answer incorrect.
Citation
@misc{hellin2026bim,
title={BIM Information Extraction Through LLM-based Adaptive Exploration},
author={Hellin et al. (2026)},
year={2026},
note={arXiv:2605.01698}
}
1---2name: ifc-bench-v2-eval3description: Tests an LLM's ability to extract, compute, and reason over heterogeneous Building Information Modeling (BIM) data (IFC files) using adaptive code execution or static baselines. It probes robustness to data heterogeneity, documentation retrieval, and tool augmentation. Use when the user wants to benchmark on ifc-bench v2, or asks about evaluating this task. Reports aggregate accuracy.4---56# ifc-bench-v2-eval78> BIM Information Extraction Through LLM-based Adaptive Exploration — Hellin et al. (2026) (arXiv:2605.01698, 2026)910## What this evaluates1112Tests an LLM's ability to extract, compute, and reason over heterogeneous Building Information Modeling (BIM) data (IFC files) using adaptive code execution or static baselines. It probes robustness to data heterogeneity, documentation retrieval, and tool augmentation.1314## Datasets1516- **ifc-bench v2** — total 1027; splits: train/dev (513), test (514); repo https://github.com/sylvainHellin/cobbie1718## Metrics1920- `aggregate accuracy` **(primary)** — range: [0, 1]21 - A non-abstained answer is rated correct only when all four substantive criteria (faithfulness, completeness, transparency, relevance) are simultaneously satisfied. Formula: accuracy = |{i: c1,i ∧ c2,i ∧ c3,i ∧ c4,i}| / N.2223## Input / output format2425**Input**: Natural-language question, reference IFC model path, and optionally pre-fetched documentation or available tools.2627**Output**: A final natural-language answer or an explicit abstention message.2829## Scoring recipe3031```python32def compute_aggregate_accuracy(predictions, gold, criteria):33 correct = 034 for i in range(len(predictions)):35 if not predictions[i].abstained:36 if all(criteria[i][c] for c in [1, 2, 3, 4]):37 correct += 138 return correct / len(predictions)39```4041## Common pitfalls4243- Abstention is distinct from system errors (crashes, timeouts), which are excluded from evaluation.44- Faithfulness thresholds are category-dependent: Category 1 requires grounding in BIM data only, while Category 4 permits stated assumptions alongside data evidence.45- The metric is deliberately conservative; a single criterion failure across the four substantive criteria marks the entire answer incorrect.4647## Evidence (verbatim from paper)4849> The primary metric reported throughout this study is aggregate accuracy: a non-abstained answer is rated correct only when all four substantive criteria (1–4) are simultaneously satisfied: accuracy = |{i: c1,i ∧ c2,i ∧ c3,i ∧ c4,i}| / N. This metric is deliberately conservative: a single criterion failure marks the answer incorrect.5051## Citation5253```bibtex54@misc{hellin2026bim,55 title={BIM Information Extraction Through LLM-based Adaptive Exploration},56 author={Hellin et al. (2026)},57 year={2026},58 note={arXiv:2605.01698}59}60```6162- arXiv: 2605.01698
qhjqhj00