Forge Manufacturing Eval

forge-manufacturing-eval

FORGE:Fine-grained Multimodal Evaluation for Manufacturing Scenarios — Jian et al. (2026) (arXiv:2604.07413, 2026)

What this evaluates

Evaluates multimodal large language models on fine-grained manufacturing tasks, including workpiece verification, surface defect inspection, and assembly verification. It probes the models' ability to combine visual grounding with domain-specific knowledge to identify anomalies or classify conditions.

Datasets

• FORGE — total ?; splits: test (-1); repo https://github.com/AI4Manufacturing/FORGE

Metrics

• exact-match accuracy (primary) — range: percent
  • Percentage of test cases where the model's predicted label exactly matches the ground-truth label. Predicted labels are extracted from free-form model responses and compared directly to the correct answer.

Input / output format

Input: Multimodal input consisting of 2D images or 3-view point cloud renderings of manufacturing components, accompanied by a task-specific query. Depending on the evaluation setting, the input may also include reference images of normal assemblies/surfaces or in-context demonstration examples.

Output: Free-form text response containing a selected option (e.g., a letter A-F for MCQs) or a surface condition classification label.

Scoring recipe

def compute_accuracy(predictions, gold_labels):
    correct = 0
    for pred, gold in zip(predictions, gold_labels):
        pred_label = extract_label(pred) # Extract letter/label from free-form text
        if pred_label == gold:
            correct += 1
    return (correct / len(gold_labels)) * 100

Common pitfalls

• Models may struggle to correctly format the required letter or label in their free-form generation, leading to false negatives in exact-match scoring.
• The evaluation distinguishes between coarse-grained (wrong workpiece type) and fine-grained (subtle model number variations) difficulties, which require different levels of domain knowledge.
• Performance varies significantly across evaluation settings (Zero-Shot vs. Ref-Cond vs. ICD), so results must be reported per setting rather than aggregated blindly.

Evidence (verbatim from paper)

We adopt exact-match accuracy as the evaluation metric. For each test case, the model's predicted MCQ letter is extracted from its free-form response and compared with the ground-truth label. Accuracy is computed as the percentage of cases where the prediction exactly matches the correct answer.

Citation

@misc{jian2026forge,
  title={FORGE:Fine-grained Multimodal Evaluation for Manufacturing Scenarios},
  author={Jian et al. (2026)},
  year={2026},
  note={arXiv:2604.07413}
}

• arXiv: 2604.07413