Model Evaluator

Model Evaluator

Overview

Rigorous model assessment is essential to deploying trustworthy ML systems. This skill covers comprehensive metrics for classification and regression, strategies for cross-validation, statistical testing, bias and fairness auditing, and A/B testing frameworks for comparing models in production.

Classification Metrics

Core Metrics

from sklearn.metrics import (
    accuracy_score, precision_score, recall_score, f1_score,
    roc_auc_score, average_precision_score, classification_report,
    confusion_matrix,
)
import numpy as np

def classification_report_full(y_true, y_pred, y_prob=None) -> dict:
    """Comprehensive classification metrics."""
    metrics = {
        "accuracy": accuracy_score(y_true, y_pred),
        "precision_macro": precision_score(y_true, y_pred, average="macro"),
        "recall_macro": recall_score(y_true, y_pred, average="macro"),
        "f1_macro": f1_score(y_true, y_pred, average="macro"),
        # ... (condensed) ...
            metrics["auc_roc_ovr"] = roc_auc_score(
                y_true, y_prob, multi_class="ovr", average="macro"
            )

    return metrics

Metric Selection Guide

Decision Framework

Is your dataset balanced (classes within 2x of each other)?
  YES -> Accuracy is meaningful, but also report F1
  NO  -> DO NOT rely on accuracy. Use these instead:
         - F1 (balanced view)
         - Average Precision (best for heavy imbalance)
         - AUC-ROC (threshold-independent ranking)

What is more costly?
  False positives (spam filter, fraud alert):
    -> Optimize for PRECISION
  False negatives (cancer screening, security):
    -> Optimize for RECALL
  Both equally bad:
    -> Optimize for F1 score

Confusion Matrix Analysis

Visualization and Interpretation

import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.metrics import confusion_matrix, ConfusionMatrixDisplay

def plot_confusion_matrix(
    y_true, y_pred,
    class_names: list[str] = None,
    normalize: str = None,
    figsize: tuple = (8, 6),
) -> plt.Figure:
    """Plot confusion matrix with detailed annotations."""
    cm = confusion_matrix(y_true, y_pred, normalize=normalize)

    fig, ax = plt.subplots(figsize=figsize)
    # ... (condensed) ...
                })
        confused_with.sort(key=lambda x: x["count"], reverse=True)
        analysis[cls]["most_confused_with"] = confused_with[:3]

    return analysis

ROC and Precision-Recall Curves

from sklearn.metrics import roc_curve, precision_recall_curve, auc

def plot_roc_pr_curves(y_true, y_prob, figsize=(14, 5)):
    """Plot ROC and Precision-Recall curves side by side."""
    fig, (ax1, ax2) = plt.subplots(1, 2, figsize=figsize)

    # ROC Curve
    fpr, tpr, roc_thresholds = roc_curve(y_true, y_prob)
    roc_auc = auc(fpr, tpr)
    ax1.plot(fpr, tpr, label=f"AUC = {roc_auc:.3f}")
    ax1.plot([0, 1], [0, 1], "k--", alpha=0.3)
    ax1.set_xlabel("False Positive Rate")
    ax1.set_ylabel("True Positive Rate")
    ax1.set_title("ROC Curve")
    # ... (condensed) ...
    ax2.set_title("Precision-Recall Curve")
    ax2.legend()

    plt.tight_layout()
    return fig

Threshold Optimization

from sklearn.metrics import balanced_accuracy_score

def find_optimal_threshold(
    y_true, y_prob,
    metric: str = "f1",
) -> tuple[float, float]:
    """Find the threshold that maximizes a given metric."""

    thresholds = np.arange(0.1, 0.95, 0.01)
    best_threshold = 0.5
    best_score = 0

    for threshold in thresholds:
        y_pred = (y_prob >= threshold).astype(int)
# ... (condensed) ...
        if score > best_score:
            best_score = score
            best_threshold = threshold

    return best_threshold, best_score

Regression Metrics

Core Regression Metrics

from sklearn.metrics import (
    mean_absolute_error, mean_squared_error, r2_score,
    mean_absolute_percentage_error, median_absolute_error,
)

def regression_report(y_true, y_pred) -> dict:
    """Comprehensive regression metrics."""
    return {
        "mae": mean_absolute_error(y_true, y_pred),
        "rmse": np.sqrt(mean_squared_error(y_true, y_pred)),
        "mse": mean_squared_error(y_true, y_pred),
        "r2": r2_score(y_true, y_pred),
        "mape": mean_absolute_percentage_error(y_true, y_pred),
        "median_ae": median_absolute_error(y_true, y_pred),
        "max_error": float(np.max(np.abs(y_true - y_pred))),
    }

Metric Interpretation

Residual Analysis

from scipy import stats as sp_stats

def plot_residual_analysis(y_true, y_pred, figsize=(14, 10)):
    """Comprehensive residual analysis plots."""
    residuals = y_true - y_pred

    fig, axes = plt.subplots(2, 2, figsize=figsize)

    # Predicted vs Actual
    axes[0, 0].scatter(y_pred, y_true, alpha=0.5, s=10)
    min_val, max_val = min(y_true.min(), y_pred.min()), max(y_true.max(), y_pred.max())
    axes[0, 0].plot([min_val, max_val], [min_val, max_val], "r--")
    axes[0, 0].set_xlabel("Predicted")
    axes[0, 0].set_ylabel("Actual")
    # ... (condensed) ...
    sp_stats.probplot(residuals, dist="norm", plot=axes[1, 1])
    axes[1, 1].set_title("QQ Plot")

    plt.tight_layout()
    return fig

Cross-Validation Strategies

Choosing the Right Strategy

from sklearn.model_selection import (
    KFold, StratifiedKFold, TimeSeriesSplit,
    GroupKFold, RepeatedStratifiedKFold,
    cross_val_score,
)

def get_cv_strategy(
    task_type: str,
    data_type: str = "standard",
    n_splits: int = 5,
    groups=None,
):
    """Select appropriate cross-validation strategy."""

    # ... (condensed) ...

    if task_type == "classification":
        return StratifiedKFold(n_splits=n_splits, shuffle=True, random_state=42)

    return KFold(n_splits=n_splits, shuffle=True, random_state=42)

Cross-Validation with Confidence Intervals

def cv_with_confidence(model, X, y, cv=5, scoring="f1") -> dict:
    """Cross-validation with confidence interval."""
    scores = cross_val_score(model, X, y, cv=cv, scoring=scoring)

    mean = scores.mean()
    std = scores.std()
    n = len(scores)
    se = std / np.sqrt(n)

    # 95% confidence interval
    ci_low = mean - 1.96 * se
    ci_high = mean + 1.96 * se

    return {
        "mean": mean,
        "std": std,
        "scores": scores.tolist(),
        "ci_95": (ci_low, ci_high),
        "n_folds": n,
    }

Statistical Model Comparison

Paired t-Test for Model Comparison

from scipy import stats

def compare_models_statistical(
    model_a_scores: list[float],
    model_b_scores: list[float],
    alpha: float = 0.05,
) -> dict:
    """Statistical comparison of two models using paired t-test."""

    t_stat, p_value = stats.ttest_rel(model_a_scores, model_b_scores)
    mean_diff = np.mean(model_a_scores) - np.mean(model_b_scores)

    return {
        "model_a_mean": np.mean(model_a_scores),
        "model_b_mean": np.mean(model_b_scores),
        "mean_difference": mean_diff,
        "t_statistic": t_stat,
        "p_value": p_value,
        "significant": p_value < alpha,
        "better_model": "A" if mean_diff > 0 else "B",
    }

McNemar's Test (for Classification)

def mcnemar_test(y_true, y_pred_a, y_pred_b) -> dict:
    """McNemar's test: are two classifiers significantly different?"""
    from statsmodels.stats.contingency_tables import mcnemar as mcnemar_fn

    correct_a = (y_pred_a == y_true)
    correct_b = (y_pred_b == y_true)

    n01 = ((~correct_a) & correct_b).sum()  # A wrong, B right
    n10 = (correct_a & (~correct_b)).sum()   # A right, B wrong

    table = [[0, n01], [n10, 0]]
    result = mcnemar_fn(table, exact=True)

    return {
        "a_right_b_wrong": int(n10),
        "a_wrong_b_right": int(n01),
        "p_value": result.pvalue,
        "significant": result.pvalue < 0.05,
    }

Bias and Fairness Assessment

Fairness Metrics

def compute_fairness_metrics(
    y_true: np.ndarray,
    y_pred: np.ndarray,
    sensitive_attr: np.ndarray,
    privileged_value=1,
    unprivileged_value=0,
) -> dict:
    """Compute fairness metrics across a sensitive attribute."""

    priv_mask = sensitive_attr == privileged_value
    unpriv_mask = sensitive_attr == unprivileged_value

    # Demographic parity
    rate_priv = y_pred[priv_mask].mean()
    # ... (condensed) ...
        "tpr_privileged": round(tpr_priv, 4),
        "tpr_unprivileged": round(tpr_unpriv, 4),
        "fpr_privileged": round(fpr_priv, 4),
        "fpr_unprivileged": round(fpr_unpriv, 4),
    }

Fairness Metric Definitions

Subgroup Analysis

import pandas as pd

def subgroup_performance(
    y_true, y_pred, y_prob,
    group_column: np.ndarray,
    group_names: dict,
) -> pd.DataFrame:
    """Compute performance metrics per subgroup."""
    results = []

    for group_val, group_name in group_names.items():
        mask = group_column == group_val
        if mask.sum() < 10:
            continue
# ... (condensed) ...
                metrics["auc_roc"] = float(roc_auc_score(y_true[mask], prob))

        results.append(metrics)

    return pd.DataFrame(results)

A/B Testing for Models

Online A/B Test Framework

import hashlib
from dataclasses import dataclass, field
from scipy import stats

@dataclass
class ModelABTest:
    """A/B test framework for comparing models in production."""
    name: str
    model_a_name: str
    model_b_name: str
    traffic_split: float = 0.5
    results_a: list = field(default_factory=list)
    results_b: list = field(default_factory=list)

    # ... (condensed) ...
            "effect_size": "small" if abs(cohens_d) < 0.5 else "medium" if abs(cohens_d) < 0.8 else "large",
            "recommendation": "B" if p_value < 0.05 and mean_b > mean_a else "A" if p_value < 0.05 else "continue_testing",
            "samples_a": len(self.results_a),
            "samples_b": len(self.results_b),
        }

Sample Size Planning

from scipy.stats import norm

def required_sample_size(
    baseline_metric: float,
    minimum_detectable_effect: float,
    alpha: float = 0.05,
    power: float = 0.8,
    metric_std: float = None,
) -> int:
    """Calculate required sample size per group for A/B test."""
    if metric_std is None:
        metric_std = np.sqrt(baseline_metric * (1 - baseline_metric))

    z_alpha = norm.ppf(1 - alpha / 2)
    z_beta = norm.ppf(power)

    n = (2 * metric_std**2 * (z_alpha + z_beta)**2) / minimum_detectable_effect**2
    return int(np.ceil(n))

Comprehensive Scoring Report

def generate_scoring_report(
    model_name: str,
    y_true, y_pred, y_prob=None,
    sensitive_attrs: dict = None,
) -> dict:
    """Generate comprehensive scoring report."""

    report = {
        "model": model_name,
        "dataset_size": len(y_true),
        "class_distribution": {int(k): int(v) for k, v in zip(*np.unique(y_true, return_counts=True))},
    }

    report["metrics"] = classification_report_full(y_true, y_pred, y_prob)
# ... (condensed) ...
            report["fairness"][attr_name] = compute_fairness_metrics(
                y_true, y_pred, attr_values
            )

    return report

Checklist

• Select metrics appropriate to the task and class distribution
• Report multiple metrics (never rely on accuracy alone)
• Analyze the confusion matrix for systematic error patterns
• Optimize the classification threshold if using probabilities
• Use proper cross-validation (stratified for classification, temporal for time series)
• Compute confidence intervals on performance estimates
• Perform statistical tests when comparing models
• Audit for bias across sensitive attributes (gender, race, age)
• Check fairness metrics (demographic parity, equal opportunity)
• Plan A/B tests with proper sample size calculations
• Document all metrics, thresholds, and decisions in a scoring report

When to Use

Use this skill when:

• Designing or implementing model evaluator solutions
• Reviewing or improving existing model evaluator approaches
• Making architectural or implementation decisions about model evaluator
• Learning model evaluator patterns and best practices
• Troubleshooting model evaluator-related issues

Do NOT use this skill when:

• The question is about a fundamentally different technology domain
• A more specific sibling skill covers the exact topic needed
• The user needs a complete hands-on tutorial rather than expert guidance

Output Format

# Model Evaluator Analysis

## Context Assessment
[Situation summary and constraints]

## Recommended Approach
[Primary recommendation with rationale]

## Implementation Steps
1. [Step with specific details]
2. [Step with specific details]
3. [Step with specific details]

## Trade-offs and Considerations
- [Key trade-off 1]
- [Key trade-off 2]

## Next Steps
- [Immediate action item]
- [Follow-up action item]

Example

Input: "Help me implement model evaluator for a medium-scale production application"

Output: A structured analysis covering current state assessment, recommended model evaluator approach with specific patterns, implementation roadmap with milestones, and risk mitigation strategies tailored to the application scale and constraints.

Edge Cases

• Legacy system integration: When model evaluator must coexist with legacy approaches, provide a gradual migration path rather than a complete rewrite
• Scale mismatch: When the solution complexity exceeds the project scale, recommend a simpler approach and note when to revisit
• Team skill gaps: When the team lacks experience with the recommended approach, include learning resources and simpler alternatives
• Conflicting requirements: When constraints conflict (e.g., performance vs. maintainability), explicitly state the trade-off and recommend based on stated priorities