# Eval Calibration

> Confidence calibration for DSPy predictions

- Skill: `j33bs/eval-calibration` (Agent Skill)
- Install (CLI): `npx skillmds@latest add j33bs/eval-calibration`
- Raw SKILL.md: https://api.skillmd.com/api/skills/j33bs/eval-calibration/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: j33bs (https://skillmd.com/u/j33bs)
- Updated: 2026-09-22
- Page: https://skillmd.com/skills/j33bs/eval-calibration

---


# Confidence Calibration

## 🎯 Trigger Conditions
Use when asked about confidence calibration, uncertainty estimation, or reliability of DSPy predictions.

## 📚 Prerequisites
- `dspy` package installed
- Calibration dataset prepared
- Confidence scoring method chosen

## 🛠️ Calibration Techniques

### 1. Temperature Scaling
```python
import torch
import torch.nn as nn

class TemperatureScaling(nn.Module):
    def __init__(self, model):
        super().__init__()
        self.model = model
        self.temperature = nn.Parameter(torch.tensor(1.0))
    
    def forward(self, inputs):
        logits = self.model(inputs)
        return logits / self.temperature
    
    def calibrate(self, devset):
        # Find optimal temperature
        best_temp = 1.0
        best_ece = float('inf')
        
        for temp in torch.linspace(0.5, 5.0, 50):
            self.temperature.data = temp
            ece = self.calculate_ece(devset)
            if ece < best_ece:
                best_ece = ece
                best_temp = temp
        
        self.temperature.data = best_temp
        return best_temp
```

### 2. Platt Scaling
```python
from sklearn.calibration import CalibratedClassifierCV

# Create calibrated predictor
def create_calibrated_predictor(base_predictor, calibration_data):
    # Extract logits
    logits = [base_predictor.predict(x).logits for x in calibration_data]
    labels = [y for _, y in calibration_data]
    
    # Calibrate
    calibrated = CalibratedClassifierCV(
        base_predictor,
        method='sigmoid',
        cv='prefit'
    )
    calibrated.fit(logits, labels)
    
    return calibrated
```

### 3. DSPy Confidence Scoring
```python
import dspy

class CalibratedPredictor(dspy.Module):
    def __init__(self, base_predictor):
        self.base_predictor = base_predictor
        self.calibration_factor = 1.0
    
    def forward(self, question):
        result = self.base_predictor(question=question)
        
        # Apply calibration
        calibrated_answer = result.answer
        calibrated_confidence = min(result.confidence * self.calibration_factor, 1.0)
        
        return {
            "answer": calibrated_answer,
            "confidence": calibrated_confidence
        }
```

### 4. Calibration Monitoring
```python
def monitor_calibration(program, devset):
    predictions = []
    for example in devset:
        result = program(**example.inputs)
        predictions.append({
            "predicted": result.answer,
            "actual": example.answer,
            "confidence": result.confidence,
            "correct": result.answer == example.answer
        })
    
    # Calculate ECE
    ece = calculate_ece(predictions)
    
    # Calculate Brier score
    brier = calculate_brier_score(predictions)
    
    return {
        "ece": ece,
        "brier": brier,
        "predictions": predictions
    }
```

## ⚠️ Pitfalls
- **Overconfidence**: Models tend to be overconfident
- **Underconfidence**: Calibration can make models too conservative
- **Dataset shift**: Calibration may not transfer to new data
- **Complexity**: Calibration adds computational overhead

## 📖 References
- [Calibration in NLP](https://arxiv.org/abs/2104.09125)
- [Temperature Scaling](https://arxiv.org/abs/1706.04525)