# Evade Eval

> Evaluates multimodal models' ability to detect evasive or deceptive content in e-commerce product listings. It probes fine-grained single-violation detection and long-context, rule-integrated reasoning across multiple overlapping policy categories. Use when the user wants to benchmark on EVADE, or asks about evaluating this task. Reports Full Accuracy.

- Skill: `qhjqhj00/evade-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/evade-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/evade-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Productivity
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/evade-eval

---


# evade-eval

> EVADE: Multimodal Benchmark for Evasive Content Detection in E-Commerce Applications — Xu et al. (2025) (arXiv:2505.17654, 2025)

## What this evaluates

Evaluates multimodal models' ability to detect evasive or deceptive content in e-commerce product listings. It probes fine-grained single-violation detection and long-context, rule-integrated reasoning across multiple overlapping policy categories.

## Datasets

- **EVADE** — total 16794; splits: test (-1)

## Metrics

- `Partial Accuracy` — range: percent
  - Percentage of samples where the model correctly identifies at least one of the true violation categories.
- `Full Accuracy` **(primary)** — range: percent
  - Percentage of samples where the model correctly identifies all true violation categories for that sample.

## Input / output format

**Input**: Multimodal product listings (images + text descriptions). Single-Violation uses short-context prompts for individual categories; All-in-One uses long-context prompts integrating multiple policy rules.

**Output**: A set of predicted violation categories from the benchmark's taxonomy. Must adhere to the constraint that 'other' (no violation) cannot be selected alongside any regulated category.

## Scoring recipe

```python
def compute_metrics(preds, golds):
    partial_correct = 0
    full_correct = 0
    for pred, gold in zip(preds, golds):
        if set(pred) & set(gold):
            partial_correct += 1
        if set(pred) == set(gold):
            full_correct += 1
    return partial_correct / len(preds), full_correct / len(preds)
```

## Common pitfalls

- Models frequently misclassify deceptive content as benign due to embedded neutral language or visual noise obscuring key text.
- Weaker models violate the implicit constraint that 'other' (no violation) cannot be selected alongside regulated categories.
- Significant gap between partial and full accuracy (>10%) indicates models capture surface features but miss critical semantic/visual details.

## Evidence (verbatim from paper)

> A critical observation across all models is the significant gap between partial accuracy and full accuracy, often exceeding 10%. To illustrate this gap, we highlight Claude-3.7-sonnet (Figure[4j]) as the best-performing closed-source model and Qwen2.5-VL-72B (Figure[4h]) as the strongest open-source VLM.

## Citation

```bibtex
@misc{xu2025evade,
  title={EVADE: Multimodal Benchmark for Evasive Content Detection in E-Commerce Applications},
  author={Xu et al. (2025)},
  year={2025},
  note={arXiv:2505.17654}
}
```

- arXiv: 2505.17654