# Risk Data Analysis

> Risk data feature analysis Skill — data preview, smart column detection, statistical description, correlation, IV, KS/AUC, feature recommendation, export Excel report and feature list

- Skill: `yao2311/risk-data-analysis` (Agent Skill, multi-file: 14 files)
- Install (CLI): `npx skillmds@latest add yao2311/risk-data-analysis`
- Raw SKILL.md: https://api.skillmd.com/api/skills/yao2311/risk-data-analysis/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Data & Analytics
- Author: yao2311 (https://skillmd.com/u/yao2311)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/yao2311/risk-data-analysis

---


# Risk Data Feature Analysis Skill

## Environment Setup

Ensure dependencies are installed:

```bash
pip install risk-data-analysis
# Or local install
# pip install -r requirements.txt && pip install -e .
```

## Pre-import

After startup, the Agent executes the following pre-imports in the terminal (subsequent steps call functions directly):

```python
from risk_data_analysis.module1_data_preview import load_data, detect_columns, save_stage1
from risk_data_analysis.module2_common import build_meta_dict, set_default_output_dir
from risk_data_analysis.module2_describe import build_describe_table, build_org_coverage_table
from risk_data_analysis.module2_correlation import calc_correlation
from risk_data_analysis.module2_iv import calc_all_iv, build_iv_segment_stats
from risk_data_analysis.module2_ks_auc import calc_ks_auc
from risk_data_analysis.module2_summary import (
    build_summary_report, export_analysis_report,
    build_feature_recommendation,
)
```

## Step 0: Name the Analysis Task

The Agent asks the user to name the task and creates a dedicated output directory.

```
Agent displays:
Please name this analysis task (press Enter for default: analysis_20240903_143000)

User input: credit_scoring_v1

Agent executes:
import os
from datetime import datetime

task_name = "credit_scoring_v1"
if not task_name.strip():
    task_name = f"analysis_{datetime.now().strftime('%Y%m%d_%H%M%S')}"

OUTPUT_DIR = f"./outputs/{task_name}"
os.makedirs(OUTPUT_DIR, exist_ok=True)
set_default_output_dir(OUTPUT_DIR)
print(f"Output directory created: {OUTPUT_DIR}")
```

> All subsequent function calls use `OUTPUT_DIR` as the `output_dir` parameter.
> After `set_default_output_dir(OUTPUT_DIR)` is called, even if `output_dir` is omitted, intermediate files will automatically land in the task directory.

---

## Overall Flow

```
[Step 0: Name task → create OUTPUT_DIR]
[Module 1: Data Preview & Column Detection (auto-start)]
  1. load_data        → terminal execution, returns (df, info_dict)
  2. detect_columns   → terminal execution, returns column_info
  3. [Interaction]    → present all detection results at once, user corrects if needed
  4. save_stage1      → terminal execution, returns meta (output_dir=OUTPUT_DIR)
  5. [Detection]      → if info columns contain org keyword, record as org_col
  6. [Interaction]    → ask user whether to start Module 2
      ├─ Has Y column ↓
      │   [Module 2 Step 1: [Interaction] menu selection → execute in order] (output_dir=OUTPUT_DIR)
      │     Option 1: build_describe_table (statistical description)
      │     Option 2: calc_correlation (correlation matrix)
      │     Option 3: calc_all_iv (IV, with missing value strategy / binning method / categorical IV interaction)
      │     Option 4: calc_ks_auc (KS/AUC)
      │        ↓
      │   [Unified export: build_summary_report + export_analysis_report] (output_dir=OUTPUT_DIR)
      │   [Feature recommendation: build_feature_recommendation → recommended_features.txt]
      │        → one Excel analysis report + feature list + final summary
      │
      └─ No Y column ↓
          [Module 2 Step 1: menu only includes options 1/2]
            [Unified export: statistical wide table → one Excel analysis report]
```

---

## Module 1: Data Preview & Column Detection

### Step 1: Load Data File + Preview

The Agent executes via terminal:

```python
df, info = load_data(file_path, encoding="utf-8")
```

The Agent presents basic info to the user (file name, file size, rows, cols, duplicate count, first 10 rows preview).

### Step 2: Smart Column Detection

```python
column_info = detect_columns(df)
```

The Agent presents detection results:
- Detected Y column candidates (e.g., target / is_default / label)
- Detected info columns (e.g., user_id / apply_date / mobile)
- Remaining columns as X columns (features)
- Org column candidates (e.g., org or low-cardinality categorical columns)

### Step 3: Interactive Confirmation of Column Roles

The Agent presents all detection results at once and asks:

```
Are the above detection results correct? Reply with corrections if needed, or "confirm" to proceed.
```

After user confirmation:

```python
meta = save_stage1(df, y_col="is_default", info_cols=["user_id"], output_dir=OUTPUT_DIR)
```

### Step 4: Detect Org Column (org_col)

If info columns contain an org keyword column, the Agent records it as `org_col`. Subsequent IV/KS/AUC calculations will use the dual perspective of full + by-org.

### Step 5: Menu-based Interaction to Start Module 2

The Agent presents a menu; the user selects multiple options at once:

```
Please select the analysis items to execute (comma-separated, e.g., 1,2,3,4):
[1] Statistical description
[2] Correlation analysis
[3] IV calculation
[4] KS/AUC calculation
```

---

## Module 2: Analysis Execution

### Option 1: Statistical Description

```python
describe_df = build_describe_table(df, meta, output_dir=OUTPUT_DIR)
```

### Option 2: Correlation Analysis

```python
correlation_results = calc_correlation(df, meta, output_dir=OUTPUT_DIR)
```

### Option 3: IV Calculation

Two-stage interaction: first confirm missing value strategy and binning method, then execute.

```python
iv_results = calc_all_iv(
    df, meta,
    binning_method="decision_tree",  # user selection
    nan_strategy="exclude",          # user selection
    calc_cat_iv=True,
    output_dir=OUTPUT_DIR,
    org_col=org_col,  # pass when org exists
)
```

### Option 4: KS/AUC Calculation

```python
ks_auc_results = calc_ks_auc(df, meta, output_dir=OUTPUT_DIR, org_col=org_col)
```

---

## Unified Export

After all selected options finish, automatically export the Excel report:

```python
summary = build_summary_report(
    describe_df=describe_df,
    iv_results=iv_results,
    ks_auc_results=ks_auc_results,
    correlation_results=correlation_results,
    org_cov_df=org_cov_df,
)

report_path = export_analysis_report(
    summary=summary,
    describe_df=describe_df,
    iv_results=iv_results,
    ks_auc_results=ks_auc_results,
    correlation_results=correlation_results,
    org_cov_df=org_cov_df,
    output_dir=OUTPUT_DIR,
)
```

## Feature Recommendation (Step 4, auto-execute)

```python
reco = build_feature_recommendation(
    describe_df=describe_df,
    iv_results=iv_results,
    correlation_results=correlation_results,
    output_dir=OUTPUT_DIR,
)
```

The Agent presents a summary to the user and reports the path of `recommended_features.txt`.

---

## Output File List

| File | Description |
|------|-------------|
| `data_analysis_report_{timestamp}.xlsx` | Excel report (up to 8 sheets) |
| `recommended_features.txt` | Recommended feature list (`columns=[...]` format) |
| `_stage1_result.parquet` | Module 1 intermediate result |
| `_stage2_describe.parquet` | Statistical intermediate result |
| `_stage2_correlation.json` | Correlation intermediate result |
| `_stage2_iv.json` | IV intermediate result |
| `_stage2_ks_auc.json` | KS/AUC intermediate result |

---

## Notes

1. All interactions happen at the conversation layer; code contains no `input()`.
2. Functions return structured dict/DataFrame; the Agent is responsible for presenting results.
3. Default thresholds: valid rate ≥ 90%, IV ≥ 0.02, |r| ≤ 0.7, org stability ≥ 70%. All adjustable via parameters.
4. Step 0 must call `set_default_output_dir(OUTPUT_DIR)` to register the task directory, ensuring intermediate data and final reports are stored together.
5. Intermediate results are persisted to disk for recovery after kernel restart.
6. Feature recommendation executes automatically with no interaction; outputs `recommended_features.txt` for downstream modeling Skills.