Tab Eval

tab-eval

The Text Anonymization Benchmark (TAB): A Dedicated Corpus and Evaluation Framework for Text Anonymization — Pilán et al. (2022) (arXiv:2202.00443, 2022)

What this evaluates

Evaluates NLP models' ability to identify and mask personally identifiable information (direct and quasi-identifiers) in legal texts while preserving non-sensitive content. It probes both privacy protection (full coverage of masking spans) and information utility (minimizing unnecessary masking of non-identifying entities).

Datasets

• TAB corpus — total ?; splits: train (-1), dev (-1), test (-1); repo https://github.com/NorskRegnesentral/text-anonymization-benchmark

Metrics

• R_{di+qi} — range: [0, 1]
  • Standard mention-level recall on all identifiers (direct + quasi), micro-averaged over annotators. Computed as true positive mentions divided by total gold mentions.
• P_{di+qi} — range: [0, 1]
  • Standard mention-level precision on all identifiers, micro-averaged over annotators. Computed as true positive mentions divided by total predicted mentions.
• ER_{di} (primary) — range: [0, 1]
  • Entity-level recall for direct identifiers. Measures the proportion of direct identifiers that are fully covered by predicted masking spans.
• ER_{qi} — range: [0, 1]
  • Entity-level recall for quasi identifiers. Measures the proportion of quasi-identifiers that are fully covered by predicted masking spans.
• WP_{di+qi} — range: [0, 1]
  • Weighted utility metric. Computes precision but weights each unnecessarily masked term by its information content, rather than treating all terms uniformly like standard precision.

Input / output format

Input: Raw text documents (e.g., ECHR court cases or biographies) containing potential personal identifiers.

Output: IOB sequence labels per token indicating whether each token should be masked (MASK) or not (NO_MASK), corresponding to DIRECT, QUASI, or NO_MASK entity types.

Scoring recipe

def score_tab(pred_spans, gold_spans):
    # pred/gold_spans: list of (start, end, type) where type in {DIRECT, QUASI, NO_MASK}
    pred_mentions = {(s,e) for s,e,t in pred_spans if t in {DIRECT, QUASI}}
    gold_mentions = {(s,e) for s,e,t in gold_spans if t in {DIRECT, QUASI}}
    R_di_qi = len(pred_mentions & gold_mentions) / len(gold_mentions) if gold_mentions else 0.0
    P_di_qi = len(pred_mentions & gold_mentions) / len(pred_mentions) if pred_mentions else 0.0
    ER_di = entity_recall(pred_spans, gold_spans, type_filter=DIRECT)
    ER_qi = entity_recall(pred_spans, gold_spans, type_filter=QUASI)
    WP_di_qi = weighted_precision(pred_spans, gold_spans)
    return {'R_di+qi': R_di_qi, 'P_di+qi': P_di_qi, 'ER_di': ER_di, 'ER_qi': ER_qi, 'WP_di+qi': WP_di_qi}

Common pitfalls

• Conflating direct and quasi identifiers in a single recall measure hides poor performance on direct identifiers, which are most harmful for privacy.
• Using standard precision/recall without weighting for information content overestimates utility preservation because it treats all masked terms equally regardless of their actual privacy risk.
• NER-based systems often mask non-identifying entities (false positives) because they rely on predefined semantic categories rather than context-specific masking decisions.

Evidence (verbatim from paper)

Presidio's results illustrate the importance of computing separate recall measures for the direct and the quasi identifiers: although the standard, mention-level recall seems relatively good at first sight (around 0.7), a closer look at the entity-level recall over direct identifiers $ER_{di}$ shows a much poorer performance (around 0.45).

Citation

@misc{pilan2022tab,
  title={The Text Anonymization Benchmark (TAB): A Dedicated Corpus and Evaluation Framework for Text Anonymization},
  author={Pilán et al. (2022)},
  year={2022},
  note={arXiv:2202.00443}
}

• arXiv: 2202.00443