# Model Monitoring

> Skill: Model Monitoring

- Skill: `sawrus/model-monitoring` (Agent Skill)
- Install (CLI): `npx skillmds@latest add sawrus/model-monitoring`
- Raw SKILL.md: https://api.skillmd.com/api/skills/sawrus/model-monitoring/raw
- Safety review: pending (external: skill-scanner PASS, skillspector PASS)
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: sawrus (https://skillmd.com/u/sawrus)
- Updated: 2026-09-21
- Page: https://skillmd.com/skills/sawrus/model-monitoring

---

# Skill: Model Monitoring

## When to load

When setting up monitoring for a deployed model or responding to drift alerts.

## Monitoring Dimensions

```
1. Operational health
   - Latency: p50, p95, p99
   - Error rate: prediction failures, input validation failures

2. Data drift (vs training baseline)
   - PSI (Population Stability Index) per feature
   - PSI > 0.2 = significant shift → retrain likely needed

3. Model quality (when labels available)
   - Accuracy metrics after ground truth arrives
   - Business outcome correlation
```

## PSI Drift Detection

```python
def calculate_psi(expected: np.ndarray, actual: np.ndarray, buckets: int = 10) -> float:
    """PSI < 0.1: stable. 0.1-0.2: monitor. > 0.2: retrain."""
    breakpoints = np.percentile(expected, np.linspace(0, 100, buckets + 1))
    exp_counts = np.clip(np.histogram(expected, breakpoints)[0] / len(expected), 1e-4, None)
    act_counts = np.clip(np.histogram(actual, breakpoints)[0] / len(actual), 1e-4, None)
    return np.sum((act_counts - exp_counts) * np.log(act_counts / exp_counts))
```