Subject Level Inference Eval

subject-level-inference-eval

Subject-level Inference for Realistic Text Anonymization Evaluation — Oh et al. (2026) (arXiv:2604.21211, 2026)

What this evaluates

This benchmark evaluates text anonymization methods by measuring both span-level masking accuracy and subject-level privacy leakage. It probes whether anonymized text successfully prevents adversarial LLMs from inferring personal identifiable information (PII) and sensitive attributes, while maintaining text utility.

Datasets

• PANORAMA — total 151; splits: test (151); repo https://github.com/maisonOP/spia.git
• TAB — total 144; splits: test (144); repo https://github.com/maisonOP/spia.git

Metrics

• Token Recall ($R_{di+qi}$) — range: [0, 1]
  • Fraction of ground-truth PII tokens correctly masked or replaced in the anonymized output at the individual mention level.
• Entity Recall ($ER_{di}$, $ER_{qi}$) — range: [0, 1]
  • Whether all spans belonging to the same direct identifier ($ER_{di}$) or quasi-identifier ($ER_{qi}$) entity are fully masked.
• 1-AAC — range: [0, 1]
  • Protection rate for the target subject only (e.g., applicant or author), calculated as 1 minus the attribute inference accuracy for that specific subject.
• CPR (primary) — range: [0, 1]
  • Coverage Protection Rate; measures protection across all subjects in the text after matching ground-truth subjects with those identified by the adversarial LLM.
• IPR — range: [0, 1]
  • Inference Protection Rate; similar to CPR but focuses on attribute inference protection across all subjects.
• Mean Utility — range: [0, 1]
  • Average of LLM-based Readability, Meaning, and ROUGE-L scores computed on the anonymized text.

Input / output format

Input: Original text containing personal identifiable information (PII) and multiple subjects.

Output: Anonymized text where PII spans are replaced with masking tokens (e.g., [PERSON]) or redacted, generated by an anonymization method and backbone.

Scoring recipe

def evaluate_span_based(gold_text, anon_text):
    gold_spans = extract_pii_spans(gold_text)
    masked_count = count_tokens_in_output(replaced_with_mask)
    return masked_count / len(gold_spans)

def evaluate_inference_based(gold_text, anon_text, adversary_llm):
    inferred_subjects = adversary_llm.identify_and_infer(anon_text)
    matched = match_subjects(extract_subjects(gold_text), inferred_subjects)
    cpr = 1 - (correct_inferences(matched) / total_subjects)
    ipr = 1 - (attribute_inference_errors(matched) / total_attributes)
    target_subject = get_target(gold_text) # applicant/author
    1_aac = 1 - attribute_inference_accuracy(matched[target_subject])
    return cpr, ipr, 1_aac

Common pitfalls

• High span-based masking scores (e.g., ER_di ≈ 1.0) do not guarantee privacy; inference attacks can still recover ~33% of attributes via context.
• Evaluating only target-subject protection (1-AAC) masks leakage to non-target individuals, as CPR/IPR often differ significantly.
• Utility metrics are averaged across readability, meaning, and ROUGE-L, which may obscure specific degradation in factual or stylistic fidelity.

Evidence (verbatim from paper)

After matching Ground Truth subjects from original text with subjects identified from anonymized text, PII-level scoring is performed to calculate CPR and IPR as defined in Section 4. Additionally, 1-AAC is reported to express AAC as a protection rate, which measures protection for the target subject only (applicant for TAB, author for PANORAMA).

Citation

@misc{oh2026subject,
  title={Subject-level Inference for Realistic Text Anonymization Evaluation},
  author={Oh et al. (2026)},
  year={2026},
  note={arXiv:2604.21211}
}

• arXiv: 2604.21211