Pii Tagging Eval

pii-tagging-eval

MAC: Multi-Agent Constitution Learning — Thareja et al. (2026) (arXiv:2603.15968, 2026)

What this evaluates

Evaluates a model's ability to identify and extract private or sensitive information spans from text across legal, healthcare, and finance domains. It probes the model's capacity for fine-grained named entity recognition under limited labeled data and domain-specific privacy definitions.

Datasets

• ECHR — total 192; splits: train (160), val (16), test (16)
• MACCROBAT — total 192; splits: train (160), val (16), test (16)
• PUPA (Finance Subset) — total 192; splits: train (160), val (16), test (16)

Metrics

• F1 (primary) — range: percent
  • Computed via exact span-level matching between predicted and ground-truth private spans. Standard for PII/NER tasks.

Input / output format

Input: Raw text documents (case documents, clinical notes, or user-assistant interactions) from legal, healthcare, or finance domains.

Output: Predicted private information spans (start/end offsets or text spans) corresponding to fine-grained privacy types.

Scoring recipe

def compute_f1(predictions, gold):
    pred_set = set(predictions)
    gold_set = set(gold)
    tp = len(pred_set & gold_set)
    fp = len(pred_set - gold_set)
    fn = len(gold_set - pred_set)
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0
    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
    return f1 * 100

Common pitfalls

• Exact span matching requires precise boundary alignment; off-by-one errors or differing tokenization can drastically reduce F1.
• The evaluation uses a fixed 192-document sample per domain to ensure consistency, which may not reflect full dataset performance.
• LLM-as-a-judge is used for agent-level analysis but not for the main benchmark metric, which relies strictly on exact span matching.

Evidence (verbatim from paper)

We report F1 computed via exact span-level matching between predicted and ground-truth private spans, which is the standard evaluation metric for this task (Sang and De Meulder, 2003).

Citation

@misc{thareja2026mac,
  title={MAC: Multi-Agent Constitution Learning},
  author={Thareja et al. (2026)},
  year={2026},
  note={arXiv:2603.15968}
}

• arXiv: 2603.15968