# AI Model Evaluation

> Evaluate ML / LLM models — accuracy, fairness, robustness, explainability, drift — with explicit metrics, subgroup analysis, and failure-mode reporting.

- Skill: `quantumquirkxyz/ai-model-evaluation` (Agent Skill)
- Install (CLI): `npx skillmds@latest add quantumquirkxyz/ai-model-evaluation`
- Raw SKILL.md: https://api.skillmd.com/api/skills/quantumquirkxyz/ai-model-evaluation/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: quantumquirkxyz (https://skillmd.com/u/quantumquirkxyz)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/quantumquirkxyz/ai-model-evaluation

---


## Contract

- **Input:** model predictions, ground truth, subgroup labels, feature data.
- **Output:** evaluation report with recommendation.
- **Side effects:** none (analysis only; does not deploy).
- **Dependencies:** model artifacts, test set, subgroup metadata.
- **Stop condition:** report complete with subgroup analysis.
- **Risk:** medium — recommendations affect deployment; requires validation.
- **Boundary:** evaluates; does not deploy.

# Model Evaluation

Evaluate a **machine-learning or LLM model** with metrics, subgroup fairness, robustness, and explainability — and recommend deployment or revision.

## Process

### 1. Define metrics
Select metrics aligned with the problem:
- Classification: accuracy, precision, recall, F1, ROC-AUC, PR-AUC, log-loss, Cohen's kappa.
- Regression: MAE, RMSE, MAPE, R², explained variance.
- Ranking / recommendation: NDCG, MAP, hit-rate, lift.
- LLM: BLEU, ROUGE, BERTScore (not just human preference).

**Completion criterion:** metrics named; justification stated.

### 2. Subgroup / fairness analysis
Split predictions by subgroup: gender, age, race, region, income, language.
Compute:
- Demographic parity (P(pred=1 | group))
- Equalised odds (TPR/FPR equality)
- Calibration (predicted probability matches observed rate)
- Performance gap (F1 difference between groups)

Report the largest gap and whether it exceeds an acceptable threshold.

**Completion criterion:** subgroup tables saved; largest gap reported.

### 3. Robustness
Test under perturbation:
- Adversarial examples (FGSM, PGD for images; word substitution for text).
- Noise (Gaussian, dropout, label noise).
- Distribution shift (test on a different domain / time period).

**Completion criterion:** robustness results with failure examples.

### 4. Explainability
- Feature importance (SHAP, permutation, mutual information).
- Counterfactual: what minimal change flips the prediction?
- Attention maps / saliency (for vision / NLP).

**Completion criterion:** explanation method applied to worst-case examples.

### 5. Failure-mode analysis
From confusion matrix or error set:
- Which cases are wrong? (false positives, false negatives, outliers)
- Is the error systematic (bias) or random?
- What does the failure reveal about model limits?

**Completion criterion:** failure-mode section with examples.

### 6. Recommendation
- Deploy if: metrics exceed thresholds; fairness gaps below threshold; robustness acceptable; failure modes understood and mitigated.
- Revise if: weak on one dimension but fixable.
- Do not deploy if: large fairness gap; brittle to shift; failure modes dangerous.

**Completion criterion:** recommendation with explicit conditions.