Primesrl Eval

primesrl-eval

PriMeSRL-Eval: A Practical Quality Metric for Semantic Role Labeling Systems Evaluation — Jindal et al. (2022) (arXiv:2210.06408, 2022)

What this evaluates

Evaluates the quality of Semantic Role Labeling (SRL) systems by measuring precision and recall for predicate senses and argument annotations. It specifically probes a model's step-dependent error propagation by penalizing argument scores when the associated predicate sense is incorrect, while also handling discontinuous and reference arguments.

Datasets

• (no dataset; pure metric skill)

Metrics

• PriMeSRL-Eval (primary) — range: [0, 1]
  • Computes Precision and Recall for both predicate senses and argument labels. It enforces strict error propagation: if a predicted predicate sense is incorrect, all associated arguments are penalized (scored as 0). The metric also unifies head and span evaluation and correctly handles discontinuous arguments and reference arguments (e.g., C-A0).

Input / output format

Input: A sentence with a target predicate word, along with gold and predicted annotations for predicate sense, argument spans, and argument labels.

Output: Precision and Recall scores for predicates and arguments, typically aggregated into an overall F1 or reported separately.

Scoring recipe

def primesrl_eval(predictions, gold):
    pred_correct = 0
    arg_correct = 0
    total_pred = len(gold.predicates)
    total_arg = len(gold.arguments)
    for p in gold.predicates:
        if p.sense == predictions[p.sense]:
            pred_correct += 1
            for arg in p.arguments:
                if arg.span == predictions[arg.span] and arg.label == predictions[arg.label]:
                    arg_correct += 1
    p_pred = pred_correct / total_pred if total_pred > 0 else 0
    r_pred = pred_correct / total_pred if total_pred > 0 else 0
    p_arg = arg_correct / total_arg if total_arg > 0 else 0
    r_arg = arg_correct / total_arg if total_arg > 0 else 0
    return p_pred, r_pred, p_arg, r_arg

Common pitfalls

• Existing metrics like CoNLL05/09 do not penalize arguments when the predicate sense is wrong, leading to systematically overestimated performance.
• Discontinuous arguments and reference arguments (e.g., C-A0) are often mishandled or ignored in standard shared-task evaluations.
• Evaluating the verb argument (predicate) separately can overestimate overall system performance since it is often trivial to identify.

Evidence (verbatim from paper)

PriMeSRL-Eval introduces a novel evaluation metric for Semantic Role Labeling (SRL) that enforces strict error propagation modeling by penalizing argument labels when predicate senses are incorrect, addressing critical flaws in existing metrics like CoNLL05 and CoNLL09. It accurately evaluates discontinuous arguments and reference arguments (e.g., C-A0), and unifies argument head and span evaluation, leading to significant performance drops and ranking shifts across state-of-the-art SRL models—demonstrating that prior evaluations systematically overestimate model quality due to lack of step-dependent error propagation.

Citation

@misc{jindal2022primesrleval,
  title={PriMeSRL-Eval: A Practical Quality Metric for Semantic Role Labeling Systems Evaluation},
  author={Jindal et al. (2022)},
  year={2022},
  note={arXiv:2210.06408}
}

• arXiv: 2210.06408