Forensics Bench Eval

forensics-bench-eval

Forensics-Bench: A Comprehensive Forgery Detection Benchmark Suite for Large Vision Language Models — Jin Wang et al. (2025) (arXiv:2503.15024, 2025)

What this evaluates

This benchmark evaluates Large Vision Language Models (LVLMs) on their ability to detect and attribute image or video forgeries. It probes generalization and reasoning capabilities across five dimensions: semantics, modalities, tasks, forgery types, and generation models, using multi-choice visual questions.

Datasets

• Forensics-Bench — total 63292; splits: test (63292)

Metrics

• accuracy (primary) — range: percent
  • Accuracy is the percentage of correctly answered multi-choice questions out of the total. Scores are averaged across the full dataset and its five sub-dimensions.

Input / output format

Input: An image paired with a multi-choice question prompt containing several options (e.g., forgery type, task, or generation model).

Output: The model's generated text response, which should contain the correct option letter or the full option text.

Scoring recipe

def compute_accuracy(predictions, gold):
    correct = 0
    for pred, g in zip(predictions, gold):
        # Step 1: Check option letter
        if any(c in pred for c in 'ABCDEFGHIJKLMNOPQRSTUVWXYZ'):
            pred_label = extract_letter(pred)
        # Step 2: Check option content
        elif any(opt in pred for opt in g['options']):
            pred_label = g['correct_option']
        # Step 3: Fallback to LLM extraction
        else:
            pred_label = llm_extract_match(pred, g['options'])
        # If extraction fails, mark as incorrect (Z)
        if pred_label == 'Z' or pred_label != g['correct_option']:
            continue
        correct += 1
    return (correct / len(predictions)) * 100

Common pitfalls

• Proprietary LVLMs often output conservative responses (e.g., 'I cannot determine...'), which are counted as incorrect and artificially lower accuracy.
• The evaluation pipeline uses a three-step fallback (letter -> content -> ChatGPT extraction) that can fail, defaulting to 'Z' and penalizing the model even if the answer was implicitly correct.
• Performance is highly skewed by forgery type; models excel at simple spoofing/style translation (>99%) but fail on complex edits like face swaps (<55%), masking overall generalization ability.

Evidence (verbatim from paper)

With the evaluation tool provided in OpenCompass, we followed previous studies to conduct evaluations: 1) we first manually check whether the option letter appears in the LVLMs’ answers; 2) we then manually check whether the option content appears in the LVLMs’ answers; 3) we finally resort ChatGPT to help extract the matching option. If the above extractions still fail, we set the model’s answer as Z. As for evaluation metrics, we use accuracy in our experiments.

Citation

@misc{wang2025forensicsbench,
  title={Forensics-Bench: A Comprehensive Forgery Detection Benchmark Suite for Large Vision Language Models},
  author={Jin Wang et al. (2025)},
  year={2025},
  note={arXiv:2503.15024}
}

• arXiv: 2503.15024