Fingerprint Persistence Eval

fingerprint-persistence-eval

FPEdit: Robust LLM Fingerprinting through Localized Parameter Editing — Wang et al. (2025) (arXiv:2508.02092, 2025)

What this evaluates

Evaluates the robustness and persistence of natural language fingerprints embedded in LLMs against downstream fine-tuning, quantization, pruning, and model merging. It also measures the initial effectiveness of fingerprint elicitation and the harmlessness to baseline model capabilities.

Datasets

• Alpaca-GPT4 — total 52000; splits: train (52000)
• ShareGPT — total 15000; splits: train (15000)
• Dolly 2 — total 15000; splits: train (15000)

Metrics

• FSR (primary) — range: percent
  • Fingerprint Survival Rate (FSR) measures the proportion of fingerprint triggers that elicit an exact match to the target fingerprint response. It is calculated as the number of exact matches divided by the total number of queries, averaged over 10 stochastic queries per trigger.
• Perplexity (PPL) — range: other
  • Cross-entropy loss per token used to measure the naturalness of fingerprint triggers. Lower PPL indicates better stealthiness against perplexity-based filters.
• Benchmark Degradation — range: percent
  • Percentage point difference in performance on 20 standard evaluation tasks before and after fingerprint embedding, measuring harmlessness.

Input / output format

Input: Natural language fingerprint trigger prompt (x)

Output: Exact target fingerprint response (y)

Scoring recipe

def compute_fsr(triggers, targets, model):
    exact_matches = 0
    total_queries = 0
    for trigger, target in zip(triggers, targets):
        for _ in range(10):
            output = model.generate(
                trigger, temperature=1.0, top_p=0.95, top_k=50
            )
            if output.strip() == target.strip():
                exact_matches += 1
            total_queries += 1
    return (exact_matches / total_queries) * 100

Common pitfalls

• FSR relies on exact string matching; minor formatting or whitespace differences between the generated output and the gold target will count as a failure.
• The metric is highly sensitive to sampling hyperparameters; the paper specifies temperature=1, top-p=0.95, and top-k=50, which must be strictly followed to reproduce the reported values.
• Persistence FSR is evaluated only after fine-tuning on three specific instruction datasets (Alpaca-GPT4, ShareGPT, Dolly 2); results will not generalize to other fine-tuning distributions or compression techniques without re-evaluation.

Evidence (verbatim from paper)

FSR measures the proportion of triggers eliciting exact target matches. ... Each model is queried with every trigger 10 times, and we report the average FSR defined in Equation[10].

Citation

@misc{wang2025fpedit,
  title={FPEdit: Robust LLM Fingerprinting through Localized Parameter Editing},
  author={Wang et al. (2025)},
  year={2025},
  note={arXiv:2508.02092}
}

• arXiv: 2508.02092