Eval Benchmark

Evaluation Benchmark

🎯 Trigger Conditions

Use when asked about creating evaluation benchmarks, testing DSPy programs, or measuring model performance.

📚 Prerequisites

• dspy package installed
• Evaluation dataset prepared
• Metrics defined

🛠️ Benchmark Creation

1. Basic Benchmark Setup

from dspy.evaluate import Evaluate
from dspy.evaluate.metrics import answer_exact_match

# Create benchmark
def create_benchmark(devset, metric):
    evaluator = Evaluate(
        devset=devset,
        metric=metric,
        num_threads=4,
        display_progress=True,
        display_table=5
    )
    return evaluator

# Run benchmark
evaluator = create_benchmark(dev_set, answer_exact_match)
results = evaluator(program)

2. Multi-Metric Benchmark

def multi_metric_benchmark(program, devset):
    metrics = {
        "exact_match": answer_exact_match,
        "semantic_similarity": semantic_similarity,
        "fluency": fluency_score
    }
    
    results = {}
    for name, metric in metrics.items():
        evaluator = Evaluate(devset=devset, metric=metric, num_threads=4)
        results[name] = evaluator(program)
    
    return results

3. Cross-Dataset Benchmark

def cross_dataset_benchmark(program, datasets):
    results = {}
    for name, devset in datasets.items():
        evaluator = Evaluate(devset=devset, metric=answer_exact_match, num_threads=4)
        results[name] = evaluator(program)
    
    return results

4. Statistical Benchmark

import statistics

def statistical_benchmark(program, devset, n_runs=10):
    scores = []
    for _ in range(n_runs):
        evaluator = Evaluate(devset=devset, metric=answer_exact_match, num_threads=4)
        results = evaluator(program)
        scores.append(results["accuracy"])
    
    return {
        "mean": statistics.mean(scores),
        "median": statistics.median(scores),
        "stdev": statistics.stdev(scores),
        "min": min(scores),
        "max": max(scores)
    }

⚠️ Pitfalls

• Dataset bias: Ensure representative data
• Metric alignment: Metrics should match objectives
• Statistical significance: Use enough runs
• Reproducibility: Fix random seeds

📖 References

• DSPy Evaluation
• Benchmark Design