Propsegment Eval

propsegment-eval

PropSegmEnt: A Large-Scale Corpus for Proposition-Level Segmentation and Entailment Recognition — Chen et al. (2022) (arXiv:2212.10750, 2022)

What this evaluates

Evaluates a model's ability to decompose sentences into atomic semantic units (propositional segmentation) and determine entailment relationships between text spans. It probes fine-grained compositional semantic alignment and partial entailment recognition beyond sentence-level NLI.

Datasets

• PropSegmEnt — total ?; splits: train (-1), test (-1); repo https://github.com/google-research-datasets/propsegment

Metrics

• Precision/Recall/F1w (macro-averaged) (primary) — range: [0, 1]
  • Macro-averaged precision and recall computed over sentences after bipartite matching predicted and gold propositions using the Hungarian algorithm. F1w is the weighted harmonic mean of precision and recall.
• Jaccard similarity (θ=0.8) — range: [0, 1]
  • Fuzzy matching threshold applied to proposition pairs; a match counts only if the Jaccard similarity between tokens is ≥ 0.8.
• Exact Match — range: [0, 1]
  • Strict token-level equality check between predicted and gold propositions.
• Balanced Accuracy — range: [0, 1]
  • Average of the true positive rate and true negative rate, used to mitigate label imbalance in two-way entailment classification.
• F1 score (per label) — range: [0, 1]
  • F1 score computed independently for each of the three entailment labels (entail, not-entail, neutral).

Input / output format

Input: For segmentation: a single input sentence. For entailment: a pair of text spans (premise and hypothesis).

Output: For segmentation: a set of predicted propositions. For entailment: a classification label (entail, not-entail, or neutral).

Scoring recipe

def score_segmentation(pred_props, gold_props):
    matches = hungarian_algorithm(pred_props, gold_props, sim=jaccard)
    tp = sum(1 for p, g in matches if jaccard(p, g) >= 0.8)
    prec = tp / len(pred_props) if pred_props else 0
    rec = tp / len(gold_props) if gold_props else 0
    f1w = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
    return prec, rec, f1w

def score_entailment(pred_labels, gold_labels):
    acc = accuracy(pred_labels, gold_labels)
    tpr = tp_rate(gold_labels, pred_labels)
    tnr = tn_rate(gold_labels, pred_labels)
    bal_acc = (tpr + tnr) / 2
    f1_per_label = f1_score(gold_labels, pred_labels, average=None)
    return acc, bal_acc, f1_per_label

Common pitfalls

• Models frequently predict repeated or redundant propositions (up to >20% for encoder+tagger baselines), requiring explicit post-processing deduplication before scoring.
• Standard accuracy is misleading due to label imbalance in the entailment task; balanced accuracy must be reported instead.
• Fuzzy matching uses a strict Jaccard threshold (θ=0.8); pairs scoring below this are treated as non-matches.
• Precision and recall are macro-averaged over sentences, not micro-averaged across all propositions.

Evidence (verbatim from paper)

Propositional Segmentation We measure the precision and recall between the set of predicted and gold propositions for a given sentence. As the set of gold propositions do not follow any particular ordering, we first produce a bipartite matching between them using the Hungarian algorithm (Kuhn, 1955). We again use the Jaccard similarity over $ heta = 0.8$ as a fuzzy match between two propositions (§ 3.2). We also use exact match, an even more restrictive measure where two propositions match if and only if they have the exact same tokens. We report the macro-averaged precision and recall over sentences in the test set.

Citation

@misc{chen2022propsegment,
  title={PropSegmEnt: A Large-Scale Corpus for Proposition-Level Segmentation and Entailment Recognition},
  author={Chen et al. (2022)},
  year={2022},
  note={arXiv:2212.10750}
}

• arXiv: 2212.10750