Piibench Eval

piibench-eval

PIIBench: A Unified Multi-Source Benchmark Corpus for Personally Identifiable Information Detection — Jha (2026) (arXiv:2604.15776, 2026)

What this evaluates

This benchmark probes the ability of NER and PII detection systems to accurately identify and classify personally identifiable information spans across highly heterogeneous, cross-domain text sources. It specifically evaluates cross-domain generalization and robustness to diverse, fine-grained PII entity types that are rarely seen together in standard training corpora.

Datasets

• PIIBench — total 2370000; splits: test (1398); repo https://github.com/pritesh-2711/pii-bench

Metrics

• span-level F1 (primary) — range: [0, 1]
  • Computed using seqeval with exact match criteria: both the character span boundaries and the entity type label must exactly match the ground truth. Precision and Recall are reported alongside F1.
• Precision — range: [0, 1]
  • Fraction of predicted spans that exactly match ground truth span boundaries and entity type labels.
• Recall — range: [0, 1]
  • Fraction of ground truth spans that are exactly matched by predicted span boundaries and entity type labels.

Input / output format

Input: Raw text sequences. Models output predicted character-offset spans, which are then mapped to the 48 canonical PIIBench entity types.

Output: BIO token labels derived from character-to-token alignment of predicted spans, or directly character-offset spans mapped to canonical PII types.

Scoring recipe

def compute_metrics(predictions, gold):
    # 1. Map predicted character offsets to BIO token labels
    aligned_preds = char_offsets_to_bio(predictions, gold_tokens)
    # 2. Discard any predicted entity types not in PIIBench's 48-type taxonomy
    aligned_preds = filter_canonical_types(aligned_preds, canonical_types)
    # 3. Compute exact-match span-level metrics using seqeval
    precision, recall, f1 = seqeval.evaluate(aligned_preds, gold_bio_labels)
    return {'precision': precision, 'recall': recall, 'f1': f1}

Common pitfalls

• Predicted entity types must be mapped to PIIBench's 48 canonical labels using a hand-constructed alignment table; types not in the taxonomy are discarded rather than treated as errors.
• Models output character-offset spans, which must be aligned to BIO token labels before evaluation; raw offsets cannot be directly compared to token-level gold standards.
• Specialized models (e.g., financial NER) often exhibit high precision but near-zero recall on mixed-domain test sets due to domain-silo effects, skewing overall F1.

Evidence (verbatim from paper)

Evaluation uses span-level seqeval metrics (Precision, Recall, F1), which require exact match of both the entity span boundaries and the entity type label. We reconstruct each system's predicted character-offset spans back to BIO token labels using a character-to-token alignment procedure, then apply seqeval to the aligned predictions.

Citation

@misc{jha2026piibench,
  title={PIIBench: A Unified Multi-Source Benchmark Corpus for Personally Identifiable Information Detection},
  author={Jha (2026)},
  year={2026},
  note={arXiv:2604.15776}
}

• arXiv: 2604.15776