# Eval Benchmark

> Benchmark creation and evaluation patterns

- Skill: `j33bs/eval-benchmark` (Agent Skill)
- Install (CLI): `npx skillmds@latest add j33bs/eval-benchmark`
- Raw SKILL.md: https://api.skillmd.com/api/skills/j33bs/eval-benchmark/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: j33bs (https://skillmd.com/u/j33bs)
- Updated: 2026-09-22
- Page: https://skillmd.com/skills/j33bs/eval-benchmark

---


# Evaluation Benchmark

## 🎯 Trigger Conditions
Use when asked about creating evaluation benchmarks, testing DSPy programs, or measuring model performance.

## 📚 Prerequisites
- `dspy` package installed
- Evaluation dataset prepared
- Metrics defined

## 🛠️ Benchmark Creation

### 1. Basic Benchmark Setup
```python
from dspy.evaluate import Evaluate
from dspy.evaluate.metrics import answer_exact_match

# Create benchmark
def create_benchmark(devset, metric):
    evaluator = Evaluate(
        devset=devset,
        metric=metric,
        num_threads=4,
        display_progress=True,
        display_table=5
    )
    return evaluator

# Run benchmark
evaluator = create_benchmark(dev_set, answer_exact_match)
results = evaluator(program)
```

### 2. Multi-Metric Benchmark
```python
def multi_metric_benchmark(program, devset):
    metrics = {
        "exact_match": answer_exact_match,
        "semantic_similarity": semantic_similarity,
        "fluency": fluency_score
    }
    
    results = {}
    for name, metric in metrics.items():
        evaluator = Evaluate(devset=devset, metric=metric, num_threads=4)
        results[name] = evaluator(program)
    
    return results
```

### 3. Cross-Dataset Benchmark
```python
def cross_dataset_benchmark(program, datasets):
    results = {}
    for name, devset in datasets.items():
        evaluator = Evaluate(devset=devset, metric=answer_exact_match, num_threads=4)
        results[name] = evaluator(program)
    
    return results
```

### 4. Statistical Benchmark
```python
import statistics

def statistical_benchmark(program, devset, n_runs=10):
    scores = []
    for _ in range(n_runs):
        evaluator = Evaluate(devset=devset, metric=answer_exact_match, num_threads=4)
        results = evaluator(program)
        scores.append(results["accuracy"])
    
    return {
        "mean": statistics.mean(scores),
        "median": statistics.median(scores),
        "stdev": statistics.stdev(scores),
        "min": min(scores),
        "max": max(scores)
    }
```

## ⚠️ Pitfalls
- **Dataset bias**: Ensure representative data
- **Metric alignment**: Metrics should match objectives
- **Statistical significance**: Use enough runs
- **Reproducibility**: Fix random seeds

## 📖 References
- [DSPy Evaluation](https://dspy-docs.vercel.app/docs/deep-dive/evaluation)
- [Benchmark Design](https://arxiv.org/abs/2010.02756)