Managing Credit Risk Models

Managing Credit Risk Models

When To Use

• Periodic validation of Probability of Default (PD), Loss Given Default (LGD), or Exposure at Default (EAD) models
• Annual or triggered model performance reviews required by internal model governance or regulatory mandate (e.g., SR 11-7, CRD/CRR, IFRS 9 ECL frameworks) [VERIFY jurisdiction-specific regulatory requirements]
• Recalibration following material portfolio shifts, macroeconomic regime changes, or post-merger data integration
• Comparing challenger models against incumbent production models before promotion
• Preparing model risk management reports for MRC/board-level review

Inputs To Gather

• Development sample and out-of-time/out-of-sample validation data — confirm vintage coverage, default definition consistency, and exclusion criteria
• Model documentation — original methodology paper, variable selection rationale, and any prior validation findings
• Current production scorecards or parameter estimates — PD term structures, LGD cure/workout assumptions, EAD CCF tables
• Realized outcome data — observed default flags, recovery cashflows, drawn/undrawn balances at default
• Portfolio segmentation — rating grades, facility types, collateral categories, geographic or industry cuts
• Regulatory and policy thresholds — minimum discrimination (e.g., Gini > 0.40), calibration tolerance bands, override rate caps [VERIFY institution-specific thresholds]

Workflow

1. Scope the validation exercise

   • Identify which model components are in scope (PD only, full PD/LGD/EAD suite, segment-level vs. portfolio-level)
   • Confirm the observation window and default/loss outcome definitions match the model's design assumptions
   • Document any data limitations or exclusions upfront

2. Assess discrimination performance

   • Compute Gini coefficient (Accuracy Ratio), AUC-ROC, and Kolmogorov-Smirnov statistic on the validation sample
   • Generate CAP (Cumulative Accuracy Profile) and ROC curves
   • Segment discrimination by key risk drivers (vintage, industry, geography) to detect pockets of weakness
   • Compare current-period metrics against development-sample benchmarks and prior validation results

3. Evaluate calibration accuracy

   • Run Binomial test, Hosmer-Lemeshow test, or traffic-light approach (Basel) across PD buckets
   • For LGD: compare predicted vs. realized loss severity by collateral type and workout path
   • For EAD: compare predicted CCFs against observed utilization at default
   • Assess calibration across economic cycles — flag if model was calibrated to benign conditions and current environment is stressed [VERIFY whether TTC vs. PIT calibration applies]

4. Evaluate stability and concentration

   • Population Stability Index (PSI) on score distributions between development and recent periods
   • Characteristic Stability Index (CSI) on key input variables
   • Herfindahl index or grade-concentration analysis to detect rating migration clustering
   • Flag PSI > 0.25 or CSI > 0.25 as material shifts requiring deeper investigation [VERIFY institution-specific PSI thresholds]

5. Stress-test and sensitivity analysis

   • Perturb key macro drivers (GDP, unemployment, HPI) and assess PD/LGD migration under stressed scenarios
   • Identify variables with outsized sensitivity — single-variable stress contributions exceeding a defined threshold
   • Cross-check stressed outputs against institution's CCAR/DFAST or ICAAP submissions if applicable [VERIFY regulatory stress testing framework]

6. Document findings and recommend actions

   • Classify findings by severity: Tier 1 (material, requires remediation before next use), Tier 2 (significant, remediation within defined timeline), Tier 3 (minor, monitor)
   • Provide specific recalibration or redevelopment recommendations with target timelines
   • Draft executive summary for Model Risk Committee or board reporting

Output

• Model Validation Report containing:
  • Executive summary with overall model rating (e.g., Satisfactory / Needs Improvement / Unsatisfactory)
  • Discrimination metrics table (Gini, AUC, KS) with trend comparison across validation periods
  • Calibration test results by grade, segment, and time horizon
  • Stability analysis with PSI/CSI tables and heatmaps
  • Findings register with severity tier, description, and remediation action/owner/deadline
  • Appendices: data quality notes, exclusion log, detailed statistical output

Quality Checks

• Confirm default and loss definitions used in validation match the model's training definitions exactly — misalignment here invalidates all downstream metrics
• Verify that validation data has no lookahead bias (outcomes must post-date the score assignment)
• Cross-check sample sizes per rating grade — bins with fewer than 30 defaults produce unreliable calibration test results
• Ensure discrimination and calibration metrics are computed on the same population; filtered vs. unfiltered samples can yield contradictory conclusions
• Validate that any override or judgmental adjustment rates are reported separately and not mixed into statistical performance metrics
• Confirm findings are mapped to specific model components — avoid blanket "model is weak" conclusions without identifying which parameter (PD, LGD, or EAD) and which segment drives the issue