Samm Eval

samm-eval

Beyond Artificial Misalignment: Detecting and Grounding Semantic-Coordinated Multimodal Manipulations — Shen et al. (2025) (arXiv:2509.12653, 2025)

What this evaluates

Detects and localizes semantically coordinated multimodal manipulations where visual edits are paired with contextually consistent textual narratives. Probes a model's ability to perform binary classification, multi-label categorization, and fine-grained visual tampering region localization using external celebrity attribute knowledge.

Datasets

• SAMM — total ?; splits: train (-1), test (-1); repo https://github.com/shen8424/SAMM-RamDG-CAP

Metrics

• ACC (primary) — range: percent
  • Standard classification accuracy calculated as the proportion of correctly predicted binary labels.
• mAP — range: percent
  • Mean Average Precision for multi-label classification tasks.
• IoUmean — range: percent
  • Mean Intersection over Union between predicted and ground-truth manipulation masks.
• F1 — range: percent
  • F1 score for tampering region detection/localization.

Input / output format

Input: Multimodal news instances comprising an image and accompanying text, potentially containing semantic-coordinated manipulations.

Output: Binary classification label, multi-label category predictions, and pixel-level masks or bounding boxes for visual tampering regions.

Scoring recipe

def compute_metrics(preds, gold):
    acc = (preds['bc'] == gold['bc']).mean() * 100
    mAP = average_precision_score(gold['mlc'], preds['mlc']) * 100
    IoUmean = compute_iou(gold['ig_mask'], preds['ig_mask']).mean() * 100
    F1 = compute_f1(gold['tg_mask'], preds['tg_mask'])
    return {'ACC': acc, 'mAP': mAP, 'IoUmean': IoUmean, 'F1': F1}

Common pitfalls

• VLLMs often achieve high classification accuracy but fail at fine-grained visual localization (e.g., FKA-Owl's mAP drops to ~1-4%).
• Performance heavily depends on retrieving correct external celebrity attribute knowledge (CAP); omitting it causes ~3% average drop.
• Generalization to unseen entities requires careful handling, as models trained on known entities may overfit to specific celebrity features.

Evidence (verbatim from paper)

Table[2] shows the performance of all the aforementioned baselines on the SAMM dataset. To simulate real-world scenarios with scarce training samples, we train these models on randomly selected subsets of 20,000 and 50,000 samples. We then evaluate their performance on the complete test set, as shown in Table[3] and Table[5]. Experimental results showed in tables prove that our method achieved state-of-the-art performance across various tasks on the SAMM dataset. Table 5. Comparison on the SAMM 50000 training set. | Methods | BC | MLC | IG | TG | | --- | --- | --- | --- | --- | | | ACC | mAP | IoUmean | F1 |

Citation

@misc{shen2025samm,
  title={Beyond Artificial Misalignment: Detecting and Grounding Semantic-Coordinated Multimodal Manipulations},
  author={Shen et al. (2025)},
  year={2025},
  note={arXiv:2509.12653}
}

• arXiv: 2509.12653