# Ml Monitoring > Monitor a live model for data quality, input and prediction drift, performance decay, and fire retraining triggers. - Skill: `matrixx0070/ml-monitoring` (Agent Skill) - Install (CLI): `npx skillmds add matrixx0070/ml-monitoring` - Raw SKILL.md: https://api.skillmd.com/api/skills/matrixx0070/ml-monitoring/raw - Safety review: pending (external: skill-scanner PASS, skillspector PASS) - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: AI & ML - Author: Matrixx0070 (https://skillmd.com/u/matrixx0070) - Updated: 2026-08-19 - Page: https://skillmd.com/skills/matrixx0070/ml-monitoring --- ## When to use Use this once a model serves real traffic, to detect when reality diverges from training — broken inputs, shifted distributions, decayed accuracy — and to define the concrete conditions that trigger retraining or rollback. **Not for:** monitoring LLM output quality, hallucination, or token cost (see `ai-guardrails`, `ai-cost-latency`). Not for BI freshness/pipeline SLAs or data-quality tests at rest (see `de-data-quality`). ## Method 1. Establish the training reference: save the training feature distributions, prediction distribution, and headline metric as the comparison baseline. 2. Monitor data quality first: nulls, out-of-range values, schema changes, unexpected categories. Most incidents are broken data, not clever drift. 3. Track input (feature) drift. Decision point: numeric → PSI or KS test; categorical → PSI / chi-square. Alert on sustained shift, not single-batch noise. 4. Track prediction drift: the score distribution moving is an early warning available before labels arrive. 5. Track true performance once labels land. Decision point: labels delayed → rely on proxy/drift signals in the gap; labels fast → monitor the live primary metric on a rolling window. 6. Segment monitoring by the cohorts that mattered in error analysis — aggregate health can hide a decayed segment. 7. Define retraining triggers as explicit thresholds (metric below X, PSI above Y, or a fixed cadence), plus a rollback trigger for hard failures. 8. Route alerts to an owner with a runbook; every alert names its likely cause and first action. ## Example Loan-default model, labels arrive ~60 days late. Daily job compares live features to the training reference: `income` PSI jumps to 0.28 (>0.2 threshold) after an upstream unit change dollars→cents — a data-quality bug, caught before any label. Prediction drift confirms the shift. Trigger fires: page the owner, roll back to v2, patch the pipeline. Standing rule: retrain when rolling AUC < 0.68 OR any feature PSI > 0.2 for 3 consecutive days OR quarterly, whichever first. ## Pitfalls - **Drift-only, no data-quality checks.** Chasing subtle distribution shifts while a null flood corrupts inputs. Check quality first. - **Waiting for labels.** Doing nothing until ground truth arrives. Use input/prediction drift as leading signals. - **Alerting on single batches.** Noisy one-off spikes cause fatigue. Require sustained breaches. - **Vague triggers.** "Retrain when it gets worse" never fires. Set numeric thresholds and a cadence. ## Output format ``` # Monitoring: REFERENCE: training feature + prediction dists + DATA QUALITY: nulls | ranges | schema | new categories INPUT DRIFT: per feature | alert= PREDICTION DRIFT: score dist | alert= PERFORMANCE: on | label delay= SEGMENTS: RETRAIN TRIGGER: Y for Nd | cadence> ROLLBACK TRIGGER: -> ALERTING: owner + runbook ```