Datanalysis Credit Risk

Credit risk data cleaning and variable screening

Execute or adapt the bundled credit-risk preprocessing pipeline without deleting original data, using independent steps to produce a cleaned modeling dataset, separated OOS samples, out-of-sample documentation, modeling/OOS feature decisions, and an Excel cleaning report.

When to invoke

• "Clean this raw credit risk dataset before modeling."
• "Run variable screening for pre-loan modeling."
• "Calculate missing rate, IV, PSI, Null Importance, and high-correlation removals."
• "Generate the credit risk cleaning Excel report."
• "Separate OOS organizations and filter abnormal months."

Prerequisites and context

• Use the bundled scripts and references in this skill package; do not delete or overwrite original input data.
• Parquet is the preferred input format for DATA_PATH when available.
• Multi-process and multi-process acceleration may be used for IV and PSI calculations when the environment supports it.
• The pipeline can prompt interactively before each step and supports default values.

Inputs

Use $ARGUMENTS as a path to the dataset, a config summary, or a request for a specific pipeline step. If $ARGUMENTS is empty, run the complete pipeline only when required parameters are already known; otherwise collect them.

Procedure

Run steps independently and preserve intermediate outputs so results can be compared.

1. Get Data: load and format raw data with get_dataset().
2. Organization Sample Analysis: compute sample count and bad sample rate per organization with org_analysis().
3. Separate OOS Data: split OOS_ORGS from modeling samples.
4. Filter Abnormal Months: remove months failing sample thresholds with drop_abnormal_ym().
5. Calculate Missing Rate: compute overall and organization-level missing rates with missing_check().
6. Drop High Missing Rate Features: remove variables above missing_ratio with drop_highmiss_features().
7. Drop Low IV Features: remove variables with weak overall IV or too many weak organizations using drop_lowiv_features().
8. Drop High PSI Features: remove unstable variables with drop_highpsi_features().
9. Null Importance Denoising: remove noise features by label permutation with drop_highnoise_features().
10. Drop High Correlation Features: remove highly correlated variables based on original gain using drop_highcorr_features().
11. Export Report: generate the Excel report with export_cleaning_report().

Quick start:

python ".github/skills/datanalysis-credit-risk/scripts/example.py"

Core functions

Screening thresholds

Report sheets

Progressive disclosure and bundled resources

• scripts/example.py: runnable example for the complete data cleaning pipeline.
• references/func.py: data loading, formatting, organization analysis, and missing-rate support functions.
• references/analysis.py: abnormal month filtering, feature screening, distributions, denoising, correlation removal, and report export functions.

Gotchas

• Do not delete original data: every step must execute independently and retain inputs for comparison.
• Do not treat OOS organizations as modeling samples: split OOS_ORGS before feature screening.
• Do not drop by missing rate alone: IV, PSI, Null Importance, and high-correlation checks remove different failure modes.
• Do not hide organization-level instability: overall quality can mask a feature that fails badly in one organization.

Output template

## Credit risk cleaning result

**Status:** complete | partial | blocked
**Data:** `DATA_PATH=<path>`
**Label/date/org:** `Y_COL=<label>`, `DATE_COL=<date>`, `ORG_COL=<organization>`
**Keys:** `KEY_COLS=<columns>`
**OOS organizations:** `OOS_ORGS=<organizations>`

### Steps
| Step | Function | Result | Features removed | Notes |
| --- | --- | --- | --- | --- |
| Get Data | `get_dataset()` | pass | 0 | <shape> |
| Drop High Missing Rate Features | `drop_highmiss_features()` | pass | <count> | `missing_ratio=0.6` |

### Report
Generated Excel report with sheets: `汇总`, `机构样本统计`, `分离OOS数据`, `Step4-异常月份处理`, `缺失率明细`, `Step5-有值率分布统计`, `Step6-高缺失率处理`, `Step7-IV明细`, `Step7-IV处理`, `Step7-IV分布统计`, `Step8-PSI明细`, `Step8-PSI处理`, `Step8-PSI分布统计`, `Step9-null importance处理`, `Step10-高相关性剔除`.

Quality gate

• DATA_PATH, DATE_COL, Y_COL, ORG_COL, KEY_COLS, and OOS_ORGS are known or explicitly reported missing.
• Original raw data is preserved and no step deletes it.
• OOS data is separated before modeling feature screening.
• Abnormal months, missing rate, IV, PSI, Null Importance, and high correlation checks were run or explicitly skipped with reason.
• Thresholds used are reported, including min_ym_bad_sample, min_ym_sample, missing_ratio, overall_iv_threshold, org_iv_threshold, max_org_threshold, psi_threshold, max_months_ratio, max_orgs, n_estimators, max_depth, gain_threshold, max_corr, and top_n_keep.
• The Excel report includes all expected sheets or documents any missing sheet.
• Bundled resources referenced above exist and are used on demand.