Built Bench Eval

built-bench-eval

Benchmarking pre-trained text embedding models in aligning built asset information — Shahinmoghadam et al. (2024) (arXiv:2411.12056, 2024)

What this evaluates

Evaluates the ability of pre-trained text embedding models to capture domain-specific semantic alignment for built asset information. The benchmark probes clustering, information retrieval, and document reranking capabilities using technical terminology from architectural, structural, mechanical, and electrical systems.

Datasets

• BuiltBench — total ?; splits: test (-1); repo https://github.com/mehrzadshm/built-bench-paper

Metrics

• task-specific metrics (primary) — range: percent
  • The paper reports scores based on task-specific metrics detailed in the original task descriptions. Clustering uses label-based similarity, retrieval uses matching accuracy/recall, and reranking uses ranking quality metrics. Scores are aggregated as averages across subtasks.

Input / output format

Input: Text strings representing built asset product names or descriptions derived from IFC and buildingSMART Data Dictionary. Queries are either short product names or longer descriptions; documents are product descriptions.

Output: Embedding vectors for each text string. For clustering: cluster assignments. For retrieval/reranking: ranked list of document indices or similarity scores.

Scoring recipe

def compute_metrics(predictions, gold):
    # Compute task-specific metrics per subtask as defined in original descriptions
    clustering_score = compute_clustering_metric(predictions, gold) * 100
    retrieval_score = compute_retrieval_metric(predictions, gold) * 100
    reranking_score = compute_reranking_metric(predictions, gold) * 100
    # Average across all six subtasks (s2s/p2p for each task)
    return (clustering_score + retrieval_score + reranking_score) / 3

Common pitfalls

• Assuming performance on general-purpose benchmarks (e.g., MTEB) transfers directly to domain-specific built asset data.
• Ignoring the significant performance drop on short-to-short (S2S) clustering tasks due to lack of contextual clues in short product names.
• Failing to apply model-specific prompts for instruction-tuned models, leading to inconsistent evaluation across base and instruction variants.

Evidence (verbatim from paper)

We applied a 1:3 positive-to-negative sampling ratio to create a balanced yet challenging evaluation set, ensuring that models must distinguish effectively between relevant and irrelevant documents. Table 2: Average scores of benchmarked models per task, based on the task-specific metrics mentioned in the task descriptions.

Citation

@misc{shahinmoghadam2024builtbench,
  title={Benchmarking pre-trained text embedding models in aligning built asset information},
  author={Shahinmoghadam et al. (2024)},
  year={2024},
  note={arXiv:2411.12056}
}

• arXiv: 2411.12056