# Tcab Eval

> Evaluates a model's ability to detect whether a given text instance has been adversarially perturbed (attack detection) and to identify the specific attack method used (attack labeling) across multiple text classification domains. Use when the user wants to benchmark on TCAB, or asks about evaluating this task. Reports balanced accuracy.

- Skill: `qhjqhj00/tcab-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/tcab-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/tcab-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/tcab-eval

---


# tcab-eval

> TCAB: A Large-Scale Text Classification Attack Benchmark — Asthana et al. (2022) (arXiv:2210.12233, 2022)

## What this evaluates

Evaluates a model's ability to detect whether a given text instance has been adversarially perturbed (attack detection) and to identify the specific attack method used (attack labeling) across multiple text classification domains.

## Datasets

- **TCAB** — total 1504607; splits: train (-1), val (-1), test (-1); repo https://github.com/REACT-NLP/tcab_generation

## Metrics

- `balanced accuracy` **(primary)** — range: [0, 1]
  - Balanced accuracy is the unweighted mean recall across all classes. For the binary attack detection task, it averages the recall for 'attack' and 'clean' classes. For the multi-class attack labeling task, it averages recall across the 12 attack methods plus the 'clean' class.

## Input / output format

**Input**: A text string (clean or adversarially perturbed) from one of six domain datasets (Climate Change, IMDB, SST-2, Wikipedia, Hatebase, Civil Comments).

**Output**: Binary label ('attack' or 'clean') for detection, or categorical label (one of 12 attack methods or 'clean') for labeling.

## Scoring recipe

```python
def balanced_accuracy(predictions, gold):
    classes = set(gold)
    recalls = []
    for c in classes:
        tp = sum(1 for p, g in zip(predictions, gold) if p == c and g == c)
        fn = sum(1 for p, g in zip(predictions, gold) if p != c and g == c)
        recalls.append(tp / (tp + fn) if (tp + fn) > 0 else 0.0)
    return sum(recalls) / len(recalls)
```

## Common pitfalls

- Data leakage: all instances derived from the same original text (clean + multiple attacks) must be kept in the same train/val/test split.
- Label preservation vs. semantic fidelity: human evaluation shows only ~50-80% of attacks preserve the original label, so detection models may be confused by semantically altered text.
- Attack success rate is not the same as detection rate: the benchmark measures the model's ability to identify attacks, not the attack's success against the victim classifier.

## Evidence (verbatim from paper)

> Table 5: Attack detection and labeling results showing the balanced accuracy of each baseline model on each dataset for attacks targeting RoBERTa. Primary Task: Attack Detection. Attack detection determines if any perturbation is present on a given piece of text [65, 42, 39, 28, 47, 16, 29]. Formally, given x* ∈ X, a model f^DET: X → {-1, +1} detects the presence of any perturbations on x*. Primary Task: Attack Labeling. The main challenge of TCAB is attack labeling — classifying which attack (if any) perturbed a given piece of text; accurate predictions on this task can provide more information about the attack/attacker.

## Citation

```bibtex
@misc{asthana2022tcab,
  title={TCAB: A Large-Scale Text Classification Attack Benchmark},
  author={Asthana et al. (2022)},
  year={2022},
  note={arXiv:2210.12233}
}
```

- arXiv: 2210.12233