Bpmn Structured Extraction Eval

bpmn-structured-extraction-eval

Structured Extraction from Business Process Diagrams Using Vision-Language Models — Deka et al. (2025) (arXiv:2511.22448, 2025)

What this evaluates

Evaluates vision-language models' ability to extract structured information (names, types, and connectivity) from Business Process Model and Notation (BPMN) diagrams provided as images. It tests both raw visual understanding and the utility of OCR-enriched inputs for schema-constrained diagram parsing.

Datasets

• BPMN Diagrams (Custom) — total ?; splits: test (-1); repo https://github.com/pritamdeka/BPMN-VLM

Metrics

• Precision — range: [0, 1]
  • Ratio of correctly predicted elements to the total number of predicted elements for a given matching criterion.
• Recall — range: [0, 1]
  • Ratio of correctly predicted elements to the total number of gold-standard elements for a given matching criterion.
• F1 Score (primary) — range: [0, 1]
  • Harmonic mean of Precision and Recall: 2 * (Precision * Recall) / (Precision + Recall). Computed across four matching levels: name-only, type-only, name+type, and relations.

Input / output format

Input: BPMN diagram images, optionally augmented with OCR-extracted text appended only when the model fails to detect key components (events, gateways, sequence flows).

Output: Structured lists of BPMN elements containing predicted names, types, and connectivity/relations.

Scoring recipe

def compute_metrics(preds, gold, criterion):
    tp = fp = fn = 0
    for p, g in zip(preds, gold):
        if matches(p, g, criterion): tp += 1
        else: fp += 1
    fn = len(gold) - tp
    prec = tp / (tp + fp) if (tp + fp) > 0 else 0
    rec = tp / (tp + fn) if (tp + fn) > 0 else 0
    f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
    return prec, rec, f1
# criterion options: 'name-only', 'type-only', 'name+type' (excludes incomplete), 'relations'

Common pitfalls

• OCR enhancement is applied conditionally (only when key components are missed), not uniformly, which can skew performance comparisons if not tracked separately.
• The name+type metric explicitly excludes incomplete entries to ensure fairness, meaning reported F1 scores may not reflect performance on partial predictions.
• Matching criteria vary significantly across settings (name-only vs. relations), so aggregating metrics without specifying the criterion leads to misleading comparisons.

Evidence (verbatim from paper)

We evaluate performance using Precision, Recall, and F1 Score, applying multiple levels of matching criteria to capture both exactness and tolerance to partial or noisy predictions. Gold-standard CSV files are constructed by parsing the corresponding .bpmn XML files for each diagram. These files provide the name, type, and relations of each BPMN element and serve as the reference for evaluation. In the name-only setting, predicted element names are compared with gold-standard names regardless of type. The type-only setting focuses exclusively on element categories, abstracting away from labels. The name+type setting requires both fields to be present and identical, with incomplete entries excluded to ensure fairness. Finally, the relations setting evaluates whether both the type and connectivity of elements align with the gold standard.

Citation

@misc{deka2025bpmnvlm,
  title={Structured Extraction from Business Process Diagrams Using Vision-Language Models},
  author={Deka et al. (2025)},
  year={2025},
  note={arXiv:2511.22448}
}

• arXiv: 2511.22448