# Model Monitoring

> Use when setting up drift detection, retraining triggers, or production model health dashboards

- Skill: `kienbui1995/model-monitoring` (Agent Skill)
- Install (CLI): `npx skillmds@latest add kienbui1995/model-monitoring`
- Raw SKILL.md: https://api.skillmd.com/api/skills/kienbui1995/model-monitoring/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: kienbui1995 (https://skillmd.com/u/kienbui1995)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/kienbui1995/model-monitoring

---


# Model Monitoring

## When to Use
When a model is live in production and needs ongoing health tracking to catch degradation before users notice.

## Core Jobs

### 1. Monitor Data Drift
Input distribution changes → model becomes unreliable.
- **Univariate drift**: monitor each feature's distribution (KS test, PSI)
- **Multivariate drift**: monitor joint distribution (MMD, PCA shift)
- Tools: Evidently AI, WhyLogs, Alibi Detect
- Threshold: PSI > 0.2 = alert, > 0.25 = investigate retraining

### 2. Monitor Prediction Drift
- Distribution of model outputs changing
- Useful when ground truth is delayed (common in production)
- Alert if output distribution shifts significantly from baseline

### 3. Monitor Ground Truth (when available)
- Compare predictions vs actuals as labels arrive
- Calculate: accuracy, precision, recall (same metrics as eval)
- Set alert thresholds: >10% degradation from baseline = page on-call

### 4. Set Retraining Triggers
Define trigger strategy:
- **Schedule**: retrain weekly/monthly regardless of drift
- **Performance-based**: retrain when accuracy drops below threshold
- **Drift-based**: retrain when PSI > 0.25 on key features
- Combine: schedule + drift detection for critical models

### 5. Dashboard Essentials
- Prediction volume over time
- Feature distributions (current vs baseline)
- Model performance metrics (rolling 7-day)
- Latency P50/P95/P99
- Error rates (failed inferences)

## Key Outputs
- Monitoring pipeline (drift detection + alerting)
- Model health dashboard
- Retraining trigger policy
- Runbook for drift alerts

## Anti-Patterns
- No monitoring after deployment ("set and forget")
- Alerting on every metric — pick 3–5 critical signals
- No ground truth pipeline — can't measure real accuracy
- Retraining without validating the new model first