Dspy Evaluation Harness

DSPy Evaluation Harness

🎯 Trigger Conditions

Use when asked about DSPy evaluation, metrics, or how to measure program quality.

📚 Prerequisites

• dspy package installed
• Evaluation dataset prepared
• Metric function defined

🛠️ Rich-Feedback Metrics

1. Basic Metrics

from dspy.evaluate import Evaluate
from dspy.evaluate.metrics import answer_exact_match, answer_passage_match

# Simple exact match
metric = answer_exact_match

# Passage matching
metric = answer_passage_match

2. Custom Feedback Metrics

import dspy

class FeedbackPrediction(dspy.Signature):
    """Evaluate the quality of an answer."""
    question = dspy.InputField()
    context = dspy.InputField()
    answer = dspy.InputField()
    correctness = dspy.OutputField(desc="0-10 score")
    explanation = dspy.OutputField(desc="Why this score?")

# Create metric function
def answer_quality(predicted, gold):
    feedback = dspy.Predict(FeedbackPrediction)(
        question=gold.question,
        context=gold.context,
        answer=predicted.answer
    )
    # Extract score
    score = int(feedback.correctness.split()[0])
    return score >= 8

# Use with Evaluate
evaluator = Evaluate(devset=dev_set, metric=answer_quality, num_threads=4)
results = evaluator(program)

3. Multi-Metric Evaluation

from dspy.evaluate.evaluate import Evaluate

def multi_metric_evaluation(predicted, gold):
    # Check exact match
    exact = answer_exact_match(predicted, gold)
    
    # Check semantic similarity
    semantic = semantic_similarity(predicted.answer, gold.answer)
    
    # Return weighted score
    return 0.7 * exact + 0.3 * semantic

evaluator = Evaluate(devset=dev_set, metric=multi_metric_evaluation, num_threads=4)

4. Statistical Analysis

import statistics

# Collect scores
scores = []
for example in dev_set:
    prediction = program(**example.inputs)
    score = metric(prediction, example)
    scores.append(score)

# Calculate statistics
mean_score = statistics.mean(scores)
median_score = statistics.median(scores)
std_score = statistics.stdev(scores)

print(f"Mean: {mean_score:.3f}")
print(f"Median: {median_score:.3f}")
print(f"StdDev: {std_score:.3f}")

⚠️ Pitfalls

• Metric alignment: Ensure metrics match your objectives
• Sample size: Use enough examples for statistical significance
• Bias: Check for dataset bias in metrics
• Overfitting: Don't optimize metrics that don't reflect quality

📖 References

• DSPy Evaluation
• Evaluation Metrics