# Eval Adversarial

> Adversarial testing and robustness evaluation

- Skill: `j33bs/eval-adversarial` (Agent Skill)
- Install (CLI): `npx skillmds@latest add j33bs/eval-adversarial`
- Raw SKILL.md: https://api.skillmd.com/api/skills/j33bs/eval-adversarial/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: j33bs (https://skillmd.com/u/j33bs)
- Updated: 2026-09-22
- Page: https://skillmd.com/skills/j33bs/eval-adversarial

---


# Adversarial Testing

## 🎯 Trigger Conditions
Use when asked about adversarial testing, robustness evaluation, or stress-testing DSPy programs.

## 📚 Prerequisites
- `dspy` package installed
- Adversarial dataset available
- Robustness metrics defined

## 🛠️ Adversarial Testing Techniques

### 1. Text Perturbation
```python
import textstat
from nltk import word_tokenize

def perturb_text(text, perturbation_type="synonym"):
    if perturbation_type == "synonym":
        # Replace words with synonyms
        words = word_tokenize(text)
        perturbed = [synonym(word) or word for word in words]
        return " ".join(perturbed)
    
    elif perturbation_type == "typo":
        # Add typos
        words = word_tokenize(text)
        perturbed = [add_typo(word) if random.random() < 0.1 else word for word in words]
        return " ".join(perturbed)
    
    elif perturbation_type == "grammar":
        # Introduce grammatical errors
        return introduce_grammar_errors(text)

# Test robustness
def test_robustness(program, test_cases):
    results = []
    for test_case in test_cases:
        original = test_case["original"]
        perturbations = [
            perturb_text(original, "synonym"),
            perturb_text(original, "typo"),
            perturb_text(original, "grammar")
        ]
        
        original_result = program(question=original)
        perturbed_results = [program(question=p) for p in perturbations]
        
        results.append({
            "original": original,
            "original_result": original_result,
            "perturbed_results": perturbed_results
        })
    
    return results
```

### 2. Adversarial Example Generation
```python
class AdversarialGenerator:
    def __init__(self, model):
        self.model = model
    
    def generate_adversarial(self, example, target_label=None):
        # Generate adversarial example
        adversarial = self.model.generate_adversarial(
            inputs=example.inputs,
            target_label=target_label
        )
        
        # Verify adversarial
        if self.is_adversarial(adversarial, example, target_label):
            return adversarial
        
        return None
    
    def is_adversarial(self, example, original, target_label):
        # Check if example causes misclassification
        prediction = self.model.predict(example)
        if target_label:
            return prediction != target_label
        else:
            return prediction != original.label
```

### 3. Stress Testing
```python
def stress_test(program, test_suite):
    results = {
        "total": len(test_suite),
        "passed": 0,
        "failed": 0,
        "errors": 0,
        "failures": []
    }
    
    for test_case in test_suite:
        try:
            result = program(**test_case.inputs)
            if test_case.metric(result, test_case.gold):
                results["passed"] += 1
            else:
                results["failed"] += 1
                results["failures"].append(test_case)
        except Exception as e:
            results["errors"] += 1
    
    return results
```

### 4. Robustness Metrics
```python
def calculate_robustness_metrics(results):
    total = len(results)
    passed = sum(1 for r in results if r["correct"])
    perturbed_passed = sum(1 for r in results if r["perturbed_correct"])
    
    return {
        "accuracy": passed / total,
        "robustness": perturbed_passed / total,
        "fragility": 1.0 - (perturbed_passed / passed) if passed > 0 else 1.0,
        "error_rate": 1.0 - (passed / total)
    }
```

## ⚠️ Pitfalls
- **Overfitting**: Programs may overfit to adversarial examples
- **Cost**: Adversarial testing is computationally expensive
- **Realism**: Artificial perturbations may not reflect real attacks
- **False positives**: Some perturbations may not actually degrade performance

## 📖 References
- [Adversarial NLP](https://arxiv.org/abs/1903.08948)
- [Robustness Evaluation](https://arxiv.org/abs/2004.06979)