qhjqhj00 copyright-tracking-tmr-eval
Tracking the Copyright of Large Vision-Language Models through Parameter Learning Adversarial Images — Wang et al. (2025) (arXiv:2502.16593, 2025)
What this evaluates
Evaluates the robustness of copyright tracking methods in fine-tuned Large Vision-Language Models (LVLMs) by measuring whether adversarial image triggers can consistently elicit a predefined target response after the model has been adapted on various downstream datasets.
Datasets
- ImageNet 2012 (validation subset) — total 200; splits: trigger_source (200)
- V7W — total 28000; splits: train (28000)
- ST-VQA — total ?; splits: train (-1)
- TextVQA — total ?; splits: train (-1)
- PaintingForm — total 20000; splits: train (20000)
- MathV360k — total 50000; splits: train (50000)
- ChEBI-20 — total ?; splits: train (-1)
Metrics
target match rate (TMR)(primary) — range: percent- Proportion of trigger queries (image-text pairs) where the model's generated response exactly matches the predefined target answer. Calculated as (matches / total queries) × 100%.
Input / output format
Input: Adversarial trigger image (perturbed from a clean ImageNet image) paired with a specific question (e.g., 'Q: Detecting copyright.').
Output: Text response generated by the LVLM.
Scoring recipe
matches = 0
total = 0
for img, question, target_answer in trigger_queries:
response = model.generate(img, question)
if response.strip() == target_answer.strip():
matches += 1
total += 1
tmr = (matches / total) * 100
Common pitfalls
- The fine-tuning datasets (V7W, ST-VQA, etc.) are used to adapt the model, not as the evaluation test set. Evaluation is strictly performed on the 1000 trigger queries.
- TMR varies significantly across different trigger QA pairs; reporting a single average without specifying the QA pair can be misleading.
- Standard adversarial attacks (Ordinary) fail post-fine-tuning due to overfitting; the method relies on parameter learning dynamics (PLA) to maintain robustness.
Evidence (verbatim from paper)
Table 1: A comprehensive comparison of our proposed method PLA with established baseline methods on the copyright tracking performance of fine-tuned models across 6 datasets. The evaluation metric is the target match rate (TMR), which measures the proportion of triggers successfully eliciting outputs that match the trigger target from the model.
Citation
@misc{wang2025tracking,
title={Tracking the Copyright of Large Vision-Language Models through Parameter Learning Adversarial Images},
author={Wang et al. (2025)},
year={2025},
note={arXiv:2502.16593}
}
- arXiv: 2502.16593