Evade Eval

evade-eval

EVADE: Multimodal Benchmark for Evasive Content Detection in E-Commerce Applications — Xu et al. (2025) (arXiv:2505.17654, 2025)

What this evaluates

Evaluates multimodal models' ability to detect evasive or deceptive content in e-commerce product listings. It probes fine-grained single-violation detection and long-context, rule-integrated reasoning across multiple overlapping policy categories.

Datasets

• EVADE — total 16794; splits: test (-1)

Metrics

• Partial Accuracy — range: percent
  • Percentage of samples where the model correctly identifies at least one of the true violation categories.
• Full Accuracy (primary) — range: percent
  • Percentage of samples where the model correctly identifies all true violation categories for that sample.

Input / output format

Input: Multimodal product listings (images + text descriptions). Single-Violation uses short-context prompts for individual categories; All-in-One uses long-context prompts integrating multiple policy rules.

Output: A set of predicted violation categories from the benchmark's taxonomy. Must adhere to the constraint that 'other' (no violation) cannot be selected alongside any regulated category.

Scoring recipe

def compute_metrics(preds, golds):
    partial_correct = 0
    full_correct = 0
    for pred, gold in zip(preds, golds):
        if set(pred) & set(gold):
            partial_correct += 1
        if set(pred) == set(gold):
            full_correct += 1
    return partial_correct / len(preds), full_correct / len(preds)

Common pitfalls

• Models frequently misclassify deceptive content as benign due to embedded neutral language or visual noise obscuring key text.
• Weaker models violate the implicit constraint that 'other' (no violation) cannot be selected alongside regulated categories.
• Significant gap between partial and full accuracy (>10%) indicates models capture surface features but miss critical semantic/visual details.

Evidence (verbatim from paper)

A critical observation across all models is the significant gap between partial accuracy and full accuracy, often exceeding 10%. To illustrate this gap, we highlight Claude-3.7-sonnet (Figure[4j]) as the best-performing closed-source model and Qwen2.5-VL-72B (Figure[4h]) as the strongest open-source VLM.

Citation

@misc{xu2025evade,
  title={EVADE: Multimodal Benchmark for Evasive Content Detection in E-Commerce Applications},
  author={Xu et al. (2025)},
  year={2025},
  note={arXiv:2505.17654}
}

• arXiv: 2505.17654