Ivy Fake Eval

ivy-fake-eval

IVY-FAKE: A Unified Explainable Framework and Benchmark for Image and Video AIGC Detection — Changjiang Jiang et al. (arXiv:2506.00979, 2025)

What this evaluates

This benchmark evaluates multimodal AI-generated content (AIGC) detection and explainable reasoning capabilities. It probes a model's ability to classify images and videos as real or fake, and to generate natural-language explanations that localize and justify synthetic artifacts.

Datasets

• Ivy-Fake — total 106000; splits: test (-1); repo https://github.com/Pi3AI/Ivy-Fake
• GenImage — total ?; splits: test (-1)
• Chameleon — total ?; splits: test (-1)
• GenVideo — total ?; splits: test (-1)

Metrics

• Accuracy (Acc) (primary) — range: [0, 1]
  • Proportion of correctly classified instances (real vs. fake) out of the total number of instances.
• Macro-averaged F1 score (F1) — range: [0, 1]
  • Harmonic mean of precision and recall calculated independently for the real and fake classes, then averaged across classes.
• ROUGE-L — range: [0, 1]
  • Longest common subsequence score between predicted and reference reasoning texts, capturing token-level overlap.
• LLM-as-a-judge score — range: [1, 5]
  • Average score (1-5) from GPT-4o mini evaluating Completeness, Relevance, Level of Detail, and Explanation. Each response is rated over five independent rounds and averaged to reduce variance.

Input / output format

Input: Image or video content (and associated prompt/question for reasoning tasks).

Output: Binary class label (real/fake) or natural language reasoning/explanation text.

Scoring recipe

def evaluate(predictions, golds, pred_texts, ref_texts):
    acc = sum(p == g for p, g in zip(predictions, golds)) / len(predictions)
    f1 = macro_f1_score(predictions, golds)
    rouge_l = rouge_l_score(pred_texts, ref_texts)
    llm_scores = []
    for text in pred_texts:
        scores = [llm_judge(prompt, text) for _ in range(5)]
        llm_scores.append(mean(scores))
    llm_score = mean(llm_scores)
    return acc, f1, rouge_l, llm_score

Common pitfalls

• ROUGE-L only measures surface-level token overlap and may miss semantic fidelity in reasoning steps.
• LLM-as-a-judge scores are sensitive to prompt phrasing and require averaging over multiple independent rounds to reduce variance.
• Video evaluation uses a many-to-many generalization setting, making cross-generator testing more complex than standard single-source benchmarks.

Evidence (verbatim from paper)

We report standard accuracy (Acc) and macro-averaged F1 score (F1) to assess the model’s ability to distinguish real from fake instances. For the reasoning task, we measure the similarity between the model’s reasoning process and the reference annotations using the ROUGE-L score*(Lin, [2004]), which captures the longest common subsequence between predicted and reference texts, reflecting token-level overlap. Since ROUGE-L may fail to fully capture the fidelity of reasoning steps, we adopt an LLM-as-a-judge evaluation paradigm(Zheng et al., [2023b]), following the FakeBench protocol(Li et al., [2024c])*, which assesses model responses along four dimensions: (1) Completeness: It reflects the extent to which the response fully addresses all aspects of the user’s question. More complete responses should incorporate information aligning well with the “golden clues” or reference answers. Incomplete or partially answered responses will receive lower scores. (2) Relevance: Measure how closely the content relates to the original annotation; (3) Level of Detail: Assess whether the response includes enough examples or elaborations; (4) Explanation: Verify the accuracy and consistency of e

Citation

@misc{jiang2025ivyfake,
  title={IVY-FAKE: A Unified Explainable Framework and Benchmark for Image and Video AIGC Detection},
  author={Changjiang Jiang et al.},
  year={2025},
  note={arXiv:2506.00979}
}

• arXiv: 2506.00979