GitHub (Microsoft) Data Cleaning and Variable Screening
Quick Start
# Run the complete data cleaning pipeline
python ".github/skills/datanalysis-credit-risk/scripts/example.py"
Complete Process Description
The data cleaning pipeline consists of the following 11 steps, each executed independently without deleting the original data:
- Get Data - Load and format raw data
- Organization Sample Analysis - Statistics of sample count and bad sample rate for each organization
- Separate OOS Data - Separate out-of-sample (OOS) samples from modeling samples
- Filter Abnormal Months - Remove months with insufficient bad sample count or total sample count
- Calculate Missing Rate - Calculate overall and organization-level missing rates for each feature
- Drop High Missing Rate Features - Remove features with overall missing rate exceeding threshold
- Drop Low IV Features - Remove features with overall IV too low or IV too low in too many organizations
- Drop High PSI Features - Remove features with unstable PSI
- Null Importance Denoising - Remove noise features using label permutation method
- Drop High Correlation Features - Remove high correlation features based on original gain
- Export Report - Generate Excel report containing details and statistics of all steps
Core Functions
| Function | Purpose | Module |
|---|---|---|
get_dataset() |
Load and format data | references.func |
org_analysis() |
Organization sample analysis | references.func |
missing_check() |
Calculate missing rate | references.func |
drop_abnormal_ym() |
Filter abnormal months | references.analysis |
drop_highmiss_features() |
Drop high missing rate features | references.analysis |
drop_lowiv_features() |
Drop low IV features | references.analysis |
drop_highpsi_features() |
Drop high PSI features | references.analysis |
drop_highnoise_features() |
Null Importance denoising | references.analysis |
drop_highcorr_features() |
Drop high correlation features | references.analysis |
iv_distribution_by_org() |
IV distribution statistics | references.analysis |
psi_distribution_by_org() |
PSI distribution statistics | references.analysis |
value_ratio_distribution_by_org() |
Value ratio distribution statistics | references.analysis |
export_cleaning_report() |
Export cleaning report | references.analysis |
Parameter Description
Data Loading Parameters
DATA_PATH: Data file path (best are parquet format)DATE_COL: Date column nameY_COL: Label column nameORG_COL: Organization column nameKEY_COLS: Primary key column name list
OOS Organization Configuration
OOS_ORGS: Out-of-sample organization list
Abnormal Month Filtering Parameters
min_ym_bad_sample: Minimum bad sample count per month (default 10)min_ym_sample: Minimum total sample count per month (default 500)
Missing Rate Parameters
missing_ratio: Overall missing rate threshold (default 0.6)
IV Parameters
overall_iv_threshold: Overall IV threshold (default 0.1)org_iv_threshold: Single organization IV threshold (default 0.1)max_org_threshold: Maximum tolerated low IV organization count (default 2)
PSI Parameters
psi_threshold: PSI threshold (default 0.1)max_months_ratio: Maximum unstable month ratio (default 1/3)max_orgs: Maximum unstable organization count (default 6)
Null Importance Parameters
n_estimators: Number of trees (default 100)max_depth: Maximum tree depth (default 5)gain_threshold: Gain difference threshold (default 50)
High Correlation Parameters
max_corr: Correlation threshold (default 0.9)top_n_keep: Keep top N features by original gain ranking (default 20)
Output Report
The generated Excel report contains the following sheets:
- 汇总 - Summary information of all steps, including operation results and conditions
- 机构样本统计 - Sample count and bad sample rate for each organization
- 分离OOS数据 - OOS sample and modeling sample counts
- Step4-异常月份处理 - Abnormal months that were removed
- 缺失率明细 - Overall and organization-level missing rates for each feature
- Step5-有值率分布统计 - Distribution of features in different value ratio ranges
- Step6-高缺失率处理 - High missing rate features that were removed
- Step7-IV明细 - IV values of each feature in each organization and overall
- Step7-IV处理 - Features that do not meet IV conditions and low IV organizations
- Step7-IV分布统计 - Distribution of features in different IV ranges
- Step8-PSI明细 - PSI values of each feature in each organization each month
- Step8-PSI处理 - Features that do not meet PSI conditions and unstable organizations
- Step8-PSI分布统计 - Distribution of features in different PSI ranges
- Step9-null importance处理 - Noise features that were removed
- Step10-高相关性剔除 - High correlation features that were removed
Features
- Interactive Input: Parameters can be input before each step execution, with default values supported
- Independent Execution: Each step is executed independently without deleting original data, facilitating comparative analysis
- Complete Report: Generate complete Excel report containing details, statistics, and distributions
- Multi-process Support: IV and PSI calculations support multi-process acceleration
- Organization-level Analysis: Support organization-level statistics and modeling/OOS distinction