Fakeclue Loki Eval

fakeclue-loki-eval

Spot the Fake: Large Multimodal Model-Based Synthetic Image Detection with Artifact Explanation — Wen et al. (2025) (arXiv:2503.14905, 2025)

What this evaluates

Evaluates large multimodal models on synthetic image detection and artifact explanation. It probes the model's ability to classify images as real or fake and generate natural language explanations for specific visual artifacts.

Datasets

• FakeClue — total ?; splits: test (-1)
• LOKI — total ?; splits: test (-1)

Metrics

• Acc (primary) — range: [0, 1]
  • Accuracy for binary real/fake classification. Calculated as the proportion of correctly classified images.
• F1 — range: [0, 1]
  • F1 score for binary real/fake classification, balancing precision and recall.
• ROUGE_L — range: [0, 1]
  • ROUGE-L metric measuring the longest common subsequence overlap between generated artifact explanations and ground truth annotations.
• CSS — range: [0, 1]
  • Custom metric for evaluating artifact explanation quality and localization, as reported in the paper.

Input / output format

Input: Image (336×336 resolution) paired with a standardized prompt: 'Does the image look real/fake?'

Output: Natural language response containing a binary classification ('Real' or 'Fake') followed by a detailed explanation of the image's artifacts.

Scoring recipe

def score(predictions, gold):
    # Detection
    pred_labels = [1 if 'fake' in p.lower() else 0 for p in predictions]
    gold_labels = [1 if 'fake' in g.lower() else 0 for g in gold]
    acc = sum(p == g for p, g in zip(pred_labels, gold_labels)) / len(gold_labels)
    f1 = f1_score(gold_labels, pred_labels)
    # Explanation
    rouge_l = rouge_score(gold, predictions, rouge_types=['rougeL'])
    css = compute_css(gold, predictions)  # Custom metric per paper
    return {'Acc': acc, 'F1': f1, 'ROUGE_L': rouge_l, 'CSS': css}

Common pitfalls

• Framing the task as a simple binary QA limits explanation quality and performance; using VQA with artifact explanations yields better alignment.
• The CSS metric is reported but its exact calculation is not detailed in the provided text, requiring reference to the appendix or code for implementation.
• Models are evaluated on both detection and explanation, but the split between general synthetic detection and DeepFake classification is not explicitly separated in the provided table.

Evidence (verbatim from paper)

The experimental results on the FakeClue and LOKI datasets include both Detection and Artifact Explanation performance. Acc ↑ F1 ↑ ROUGE_L ↑ CSS ↑

Citation

@misc{wen2025spotthefake,
  title={Spot the Fake: Large Multimodal Model-Based Synthetic Image Detection with Artifact Explanation},
  author={Wen et al. (2025)},
  year={2025},
  note={arXiv:2503.14905}
}

• arXiv: 2503.14905