Eval Calibration

Confidence Calibration

🎯 Trigger Conditions

Use when asked about confidence calibration, uncertainty estimation, or reliability of DSPy predictions.

📚 Prerequisites

• dspy package installed
• Calibration dataset prepared
• Confidence scoring method chosen

🛠️ Calibration Techniques

1. Temperature Scaling

import torch
import torch.nn as nn

class TemperatureScaling(nn.Module):
    def __init__(self, model):
        super().__init__()
        self.model = model
        self.temperature = nn.Parameter(torch.tensor(1.0))
    
    def forward(self, inputs):
        logits = self.model(inputs)
        return logits / self.temperature
    
    def calibrate(self, devset):
        # Find optimal temperature
        best_temp = 1.0
        best_ece = float('inf')
        
        for temp in torch.linspace(0.5, 5.0, 50):
            self.temperature.data = temp
            ece = self.calculate_ece(devset)
            if ece < best_ece:
                best_ece = ece
                best_temp = temp
        
        self.temperature.data = best_temp
        return best_temp

2. Platt Scaling

from sklearn.calibration import CalibratedClassifierCV

# Create calibrated predictor
def create_calibrated_predictor(base_predictor, calibration_data):
    # Extract logits
    logits = [base_predictor.predict(x).logits for x in calibration_data]
    labels = [y for _, y in calibration_data]
    
    # Calibrate
    calibrated = CalibratedClassifierCV(
        base_predictor,
        method='sigmoid',
        cv='prefit'
    )
    calibrated.fit(logits, labels)
    
    return calibrated

3. DSPy Confidence Scoring

import dspy

class CalibratedPredictor(dspy.Module):
    def __init__(self, base_predictor):
        self.base_predictor = base_predictor
        self.calibration_factor = 1.0
    
    def forward(self, question):
        result = self.base_predictor(question=question)
        
        # Apply calibration
        calibrated_answer = result.answer
        calibrated_confidence = min(result.confidence * self.calibration_factor, 1.0)
        
        return {
            "answer": calibrated_answer,
            "confidence": calibrated_confidence
        }

4. Calibration Monitoring

def monitor_calibration(program, devset):
    predictions = []
    for example in devset:
        result = program(**example.inputs)
        predictions.append({
            "predicted": result.answer,
            "actual": example.answer,
            "confidence": result.confidence,
            "correct": result.answer == example.answer
        })
    
    # Calculate ECE
    ece = calculate_ece(predictions)
    
    # Calculate Brier score
    brier = calculate_brier_score(predictions)
    
    return {
        "ece": ece,
        "brier": brier,
        "predictions": predictions
    }

⚠️ Pitfalls

• Overconfidence: Models tend to be overconfident
• Underconfidence: Calibration can make models too conservative
• Dataset shift: Calibration may not transfer to new data
• Complexity: Calibration adds computational overhead

📖 References

• Calibration in NLP
• Temperature Scaling