# Dspy Evaluation Harness

> Rich feedback metrics and evaluation patterns

- Skill: `j33bs/dspy-evaluation-harness` (Agent Skill)
- Install (CLI): `npx skillmds@latest add j33bs/dspy-evaluation-harness`
- Raw SKILL.md: https://api.skillmd.com/api/skills/j33bs/dspy-evaluation-harness/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: j33bs (https://skillmd.com/u/j33bs)
- Updated: 2026-09-22
- Page: https://skillmd.com/skills/j33bs/dspy-evaluation-harness

---


# DSPy Evaluation Harness

## 🎯 Trigger Conditions
Use when asked about DSPy evaluation, metrics, or how to measure program quality.

## 📚 Prerequisites
- `dspy` package installed
- Evaluation dataset prepared
- Metric function defined

## 🛠️ Rich-Feedback Metrics

### 1. Basic Metrics
```python
from dspy.evaluate import Evaluate
from dspy.evaluate.metrics import answer_exact_match, answer_passage_match

# Simple exact match
metric = answer_exact_match

# Passage matching
metric = answer_passage_match
```

### 2. Custom Feedback Metrics
```python
import dspy

class FeedbackPrediction(dspy.Signature):
    """Evaluate the quality of an answer."""
    question = dspy.InputField()
    context = dspy.InputField()
    answer = dspy.InputField()
    correctness = dspy.OutputField(desc="0-10 score")
    explanation = dspy.OutputField(desc="Why this score?")

# Create metric function
def answer_quality(predicted, gold):
    feedback = dspy.Predict(FeedbackPrediction)(
        question=gold.question,
        context=gold.context,
        answer=predicted.answer
    )
    # Extract score
    score = int(feedback.correctness.split()[0])
    return score >= 8

# Use with Evaluate
evaluator = Evaluate(devset=dev_set, metric=answer_quality, num_threads=4)
results = evaluator(program)
```

### 3. Multi-Metric Evaluation
```python
from dspy.evaluate.evaluate import Evaluate

def multi_metric_evaluation(predicted, gold):
    # Check exact match
    exact = answer_exact_match(predicted, gold)
    
    # Check semantic similarity
    semantic = semantic_similarity(predicted.answer, gold.answer)
    
    # Return weighted score
    return 0.7 * exact + 0.3 * semantic

evaluator = Evaluate(devset=dev_set, metric=multi_metric_evaluation, num_threads=4)
```

### 4. Statistical Analysis
```python
import statistics

# Collect scores
scores = []
for example in dev_set:
    prediction = program(**example.inputs)
    score = metric(prediction, example)
    scores.append(score)

# Calculate statistics
mean_score = statistics.mean(scores)
median_score = statistics.median(scores)
std_score = statistics.stdev(scores)

print(f"Mean: {mean_score:.3f}")
print(f"Median: {median_score:.3f}")
print(f"StdDev: {std_score:.3f}")
```

## ⚠️ Pitfalls
- **Metric alignment**: Ensure metrics match your objectives
- **Sample size**: Use enough examples for statistical significance
- **Bias**: Check for dataset bias in metrics
- **Overfitting**: Don't optimize metrics that don't reflect quality

## 📖 References
- [DSPy Evaluation](https://dspy-docs.vercel.app/docs/deep-dive/evaluation)
- [Evaluation Metrics](https://dspy-docs.vercel.app/docs/deep-dive/evaluation/metrics)