Pet Eval

pet-eval

PET: An Annotated Dataset for Process Extraction from Natural Language Text — Bellan et al. (2022) (arXiv:2203.04860, 2022)

What this evaluates

This benchmark evaluates the capability of NLP models to extract structured business process elements and their relationships from unstructured natural language text. It specifically probes entity recognition (activities, actors, gateways, data) and relation detection (flow, usage, performer/recipient) under varying information availability assumptions.

Datasets

• PET — total 45; splits: full (45)

Metrics

• F1 (primary) — range: [0, 1]
  • Harmonic mean of Precision and Recall: F1 = 2 * (Precision * Recall) / (Precision + Recall). Precision is the proportion of correctly predicted instances among all predicted instances, and Recall is the proportion of correctly predicted instances among all actual instances.

Input / output format

Input: Raw natural language text containing business process descriptions.

Output: Structured annotations for entities (activities, actors, gateways, data elements) and relations (sequence flow, uses, actor performer/recipient, further specification, same gateway). CRF baselines use IOB2 schema encoding; rule-based baselines output detected relations based on text position.

Scoring recipe

def compute_metrics(predictions, gold):
    tp = len(predictions & gold)
    fp = len(predictions - gold)
    fn = len(gold - predictions)
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0.0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0.0
    f1 = 2 * (precision * recall) / (precision + recall) if (precision + recall) > 0 else 0.0
    return precision, recall, f1

Common pitfalls

• Performance varies significantly across entity types; AND Gateways and Further Specifications show near-zero recall despite high precision.
• Evaluation settings differ based on information availability: Baseline 1 uses raw text only, while Baselines 2 and 3 leverage gold entity annotations to detect relations, making direct comparison across baselines challenging.
• The dataset uses 5-fold cross-validation rather than a fixed train/test split, so reported scores are averages over folds.

Evidence (verbatim from paper)

Table 4 provides the results obtained by the three baseline approaches described above. An observation of baselines' performance highlights a general capability of the adopted approaches in detecting both process elements and relations with a high precision. Exceptions are the further specification and AND Gateway elements, for which the baseline obtained very poor performance. While on the one hand the observed precision is high, on the other hand the recall is the metric for which lower performance were obtained. In turn, this affected the value of the F1 as well.

Citation

@misc{bellan2022pet,
  title={PET: An Annotated Dataset for Process Extraction from Natural Language Text},
  author={Bellan et al. (2022)},
  year={2022},
  note={arXiv:2203.04860}
}

• arXiv: 2203.04860