Pii Masking Eval

pii-masking-eval

Unmasking the Reality of PII Masking Models: Performance Gaps and the Call for Accountability — Singh et al. (2025) (arXiv:2504.12308, 2025)

What this evaluates

Evaluates the ability of PII masking models to correctly identify and classify sensitive information in text. It probes performance across diverse contexts, multilingual inputs, noisy formats, and evolving entity types.

Datasets

• PII Masking Dataset — total 17000; splits: test (17000)

Metrics

• non-identification (primary) — range: percent
  • Percentage of test instances where the model outputs an empty list, indicating it failed to detect any PII in the input text.
• misclassification — range: percent
  • Percentage of test instances where the model correctly detected the PII substring but assigned an incorrect PII type label.

Input / output format

Input: Text paragraphs containing a known 'seed PII' entity, with variations in phrasing, syntax, context, and multilingualism.

Output: List of substrings identified as PII along with their predicted PII type labels.

Scoring recipe

def evaluate(predictions, gold_seed, gold_type):
    detected = False
    detected_type = None
    for substr, ptype in predictions:
        if substr == gold_seed:
            detected = True
            detected_type = ptype
            break
    if not detected:
        return 'non_identified'
    elif detected_type != gold_type:
        return 'misclassified'
    return 'correct'

Common pitfalls

• Regex matching is used for string comparison, so minor formatting differences (e.g., spaces, hyphens) in the model's output vs. the seed PII will count as non-identification.
• The evaluation only checks if the exact seed PII string is present in the predictions, ignoring other correctly detected entities in the same paragraph.

Evidence (verbatim from paper)

We used regex matching between the prediction and the seed PII to find out whether the model was able to detect the correct string or not. Table 3 lists the numbers of non-identification of any PII in the input text for the models across all the feature dimensions.

Citation

@misc{singh2025unmasking,
  title={Unmasking the Reality of PII Masking Models: Performance Gaps and the Call for Accountability},
  author={Singh et al. (2025)},
  year={2025},
  note={arXiv:2504.12308}
}

• arXiv: 2504.12308