Construction Site 10k Eval

construction-site-10k-eval

Are Large Pre-trained Vision Language Models Effective Construction Safety Inspectors? — Chen et al. (2025) (arXiv:2508.11011, 2025)

What this evaluates

Evaluates vision-language models on construction site safety inspection tasks, including image captioning, safety rule violation detection, reasoning, and visual grounding of specific objects.

Datasets

• ConstructionSite 10k — total 10013; splits: test (-1)

Metrics

• IoU (primary) — range: [0, 1]
  • Intersection over Union: Area of overlap between predicted and ground-truth bounding boxes divided by their union area.
• SPICE — range: [0, 1]
  • Scene Graph Informed Caption Evaluation: Parses captions into scene graphs (objects, attributes, relationships) and computes F1 score of matching tuples.
• Precision — range: [0, 1]
  • True Positives / (True Positives + False Positives) for multi-label rule selection.
• Recall — range: [0, 1]
  • True Positives / (True Positives + False Negatives) for multi-label rule selection.
• LLM-Judge Score — range: [0, 6]
  • Sum of three criteria (Relevance, Equivalence, Specificity) scored 0-2 by a Llama 3 8B judge, max total 6.

Input / output format

Input: Image + system prompt ('You are a construction site inspector...') + user prompt (task-specific, e.g., safety rule question with 4 rules to choose from, requesting rule ID, explanation, and bounding box)

Output: For captioning: a single-paragraph text description. For VQA: a selected violated rule ID, a textual explanation, and a bounding box coordinate.

Scoring recipe

def score(preds, golds):
    # IoU for grounding
    iou = overlap(preds.box, golds.box) / union(preds.box, golds.box)
    # Precision/Recall for rule selection
    tp = sum(p == g == 1 for p, g in zip(preds.rules, golds.rules))
    fp = sum(p == 1 and g == 0 for p, g in zip(preds.rules, golds.rules))
    fn = sum(p == 0 and g == 1 for p, g in zip(preds.rules, golds.rules))
    prec = tp / (tp + fp) if (tp + fp) > 0 else 0.0
    rec = tp / (tp + fn) if (tp + fn) > 0 else 0.0
    # LLM Judge for reasoning (only if rule correctly selected)
    judge = llm_judge.evaluate(preds.reasoning, golds.reasoning, criteria=['relevance', 'equivalence', 'specificity'])
    return {'IoU': iou, 'Precision': prec, 'Recall': rec, 'LLM_Judge': judge}

Common pitfalls

• IoU is only computed for correctly identified violations in the VQA task, meaning false positives in rule selection are not penalized in the grounding score.
• Few-shot in-context learning differs between GPT models (5 images+captions) and LLaVA (5 captions-only), making cross-model comparison sensitive to modality alignment.
• LLM judge scores depend heavily on the 3-shot examples and beam search settings (k=5, seed=20) used during evaluation.

Evidence (verbatim from paper)

The evaluation metrics used for the image captioning task are: SPICE, CIDEr-D, METEOR, BERTScore, and CLIPScore... For multi-label classification (i.e. choosing the violated rules), the evaluation metrics are precision and recall for each rule. For the visual grounding, we use IoU.

Citation

@misc{chen2025construction,
  title={Are Large Pre-trained Vision Language Models Effective Construction Safety Inspectors?},
  author={Chen et al. (2025)},
  year={2025},
  note={arXiv:2508.11011}
}

• arXiv: 2508.11011