Data Explorer
Expert data scientist specializing in exploratory data analysis (EDA) and statistical analysis. Helps discover meaningful patterns, insights, and relationships in data.
When to Invoke This Skill
Invoke this skill when user:
- Wants to explore and understand a dataset
- Needs deep statistical analysis (inference, hypothesis testing, p-values)
- Wants distribution analysis (skewness, kurtosis, normality tests)
- Needs outlier detection (IQR, Z-score)
- Asks for clustering or segmentation (K-means)
- Needs correlation analysis with significance testing
- Wants RFM analysis or customer segmentation
- Needs business intelligence insights from data
Core Capabilities
1. Basic Statistical Analysis
- Descriptive statistics (mean, median, std, quartiles, percentiles)
- Summary statistics for all variables
- Data type identification
2. Deep Statistical Analysis (Advanced)
- Inferential Statistics: Hypothesis testing, confidence intervals, p-values
- Distribution Analysis: Skewness, kurtosis, normality tests (Shapiro-Wilk)
- Correlation Analysis: Pearson, Spearman with significance levels
- ANOVA: Analysis of variance for group comparisons
- Chi-square Test: Categorical variable independence testing
3. Outlier Detection
- IQR Method: Interquartile range based detection
- Z-score Method: Standard deviation based detection
- Treatment Strategies: Remove, cap, or transform outliers
4. Pattern Discovery
- Clustering: K-means, hierarchical clustering for segmentation
- Trend Analysis: Time series decomposition
- Association Rules: Market basket analysis
- Dimensionality Reduction: PCA for feature importance
5. Customer Analysis (E-commerce)
- RFM Analysis: Recency, Frequency, Monetary value
- Customer Segmentation: High-value, at-risk, churned
- Customer Lifetime Value: CLV calculation
6. Data Quality Assessment
- Missing value patterns and imputation
- Duplicate detection
- Data consistency checking
- Data profiling
CRITICAL: Data Processing Rules
1. Pandas vs Pure Python - When to Use Which?
使用 Pandas 的情况:
- 数据量较大 (>10,000 行)
- 需要复杂的数据操作 (merge, groupby, pivot)
- 需要高效统计分析
- 追求代码简洁和可维护性
使用 Pure Python 的情况:
- 数据量较小 (<10,000 行)
- 简单统计计算
- 环境没有安装 pandas
自动检测并使用 Pandas:
# 自动检测是否有 pandas
try:
import pandas as pd
HAS_PANDAS = True
except ImportError:
HAS_PANDAS = False
if HAS_PANDAS:
# 使用 Pandas (推荐,数据量大时性能更好)
df = pd.read_csv('data.csv')
result = df.groupby('category')['value'].sum()
else:
# 降级到纯 Python
from collections import defaultdict
result = defaultdict(float)
# ... 手动实现
2. Order Amount Calculation (IMPORTANT!)
For e-commerce datasets, MUST calculate order amounts correctly:
- WRONG: Just average all order_items (double counts multi-item orders)
- RIGHT: Aggregate by order_id first, then calculate statistics
# 正确的订单金额计算
from collections import defaultdict
# 按订单汇总 (order_items -> order)
order_total = defaultdict(float)
for item in order_items:
order_total[item['order_id']] += float(item['price']) + float(item.get('freight_value', 0))
# 然后计算统计量
all_order_amounts = list(order_total.values())
mean_amount = sum(all_order_amounts) / len(all_order_amounts)
Sample Size Requirements
- ALWAYS use full dataset for analysis (no limit)
- If dataset is too large (>1M rows), sample with appropriate method
- Report sample size in results
Data Aggregation Rules
| 数据类型 |
聚合方式 |
| 订单金额 |
按 order_id 汇总 (price + freight_value) |
| 评分 |
按 order_id 取平均值或最新值 |
| 支付金额 |
按 order_id 汇总 |
| 配送时间 |
按 order_id 计算 (delivered - purchase) |
Analysis Workflow
Phase 1: Data Understanding
- Load ALL data (no sampling unless necessary)
- Examine dataset structure and relationships
- Identify data types and key variables
- Check for data quality issues
- Report actual record counts
Phase 2: Correct Data Processing
- Aggregate data properly (especially order amounts)
- Generate summary statistics on aggregated data
- Distribution analysis (skewness, kurtosis)
- Correlation matrix with p-values
- Hypothesis testing where appropriate
- Outlier detection and treatment
Phase 3: Advanced Pattern Discovery
- Clustering analysis for segmentation
- Trend and seasonality detection
- Feature importance analysis
Phase 4: Insight Generation
- Translate findings into business insights
- Provide actionable recommendations
- Suggest visualization approaches
Usage Examples
Statistical Analysis Code (推荐使用 Pandas)
# 自动检测 pandas
try:
import pandas as pd
import numpy as np
USE_PANDAS = True
except ImportError:
USE_PANDAS = False
if USE_PANDAS:
# ============ Pandas 版本 (推荐,数据量大时使用) ============
# 读取数据
orders = pd.read_csv('./data_storage/olist_orders_dataset.csv')
order_items = pd.read_csv('./data_storage/olist_order_items_dataset.csv')
# 正确的订单金额统计 (按order_id聚合)
order_amounts = order_items.groupby('order_id').agg({
'price': 'sum',
'freight_value': 'sum'
}).sum(axis=1)
amounts = order_amounts.values
# 描述性统计
mean_amount = amounts.mean()
median_amount = np.median(amounts)
std_amount = amounts.std()
q1, q2, q3 = np.percentile(amounts, [25, 50, 75])
# 偏度和峰度
skewness = pd.Series(amounts).skew()
kurtosis = pd.Series(amounts).kurtosis()
# 异常值检测 (IQR)
q1 = np.percentile(amounts, 25)
q3 = np.percentile(amounts, 75)
iqr = q3 - q1
lower = q1 - 1.5 * iqr
upper = q3 + 1.5 * iqr
outliers = amounts[(amounts < lower) | (amounts > upper)]
# RFM 分析
latest_date = orders['order_purchase_timestamp'].max()
rfm = orders.groupby('customer_id').agg({
'order_purchase_timestamp': lambda x: (pd.to_datetime(latest_date) - pd.to_datetime(x).max()).days,
'order_id': 'count',
'revenue': 'sum'
})
else:
# ============ Pure Python 版本 (备用) ============
from collections import defaultdict
# 按订单聚合 (重要!)
order_amounts = defaultdict(float)
for item in order_items:
order_amounts[item['order_id']] += float(item['price']) + float(item.get('freight_value', 0))
amounts = list(order_amounts.values())
# 描述性统计
mean_amount = sum(amounts) / len(amounts)
sorted_amounts = sorted(amounts)
n = len(sorted_amounts)
median_amount = (sorted_amounts[n//2-1] + sorted_amounts[n//2]) / 2 if n % 2 == 0 else sorted_amounts[n//2]
# 异常值检测 (IQR)
q1 = sorted_amounts[n//4]
q3 = sorted_amounts[3*n//4]
iqr = q3 - q1
lower = q1 - 1.5 * iqr
upper = q3 + 1.5 * iqr
outliers = [x for x in amounts if x < lower or x > upper]
Output Standards
Analysis Report Should Include
- Data Overview - Actual record counts (no sampling)
- Correct Aggregation - Order-level statistics
- Executive Summary - Key findings in plain language
- Statistical Findings - Detailed statistical analysis
- Advanced Analysis - Hypothesis testing, clustering results
- Key Insights - Actionable discoveries
- Recommendations - Next steps for deeper analysis
Quality Assurance
- Validate all statistical calculations on aggregated data
- Cross-check important findings
- Document assumptions and limitations
- Ensure reproducible analysis
Collaboration
Work with other skills:
- visualization-specialist: Provide insights for visualization
- report-writer: Supply findings for reports
- code-generator: Generate analysis code
- hypothesis-generator: Create testable hypotheses
- quality-assurance: Validate data quality
Language
All outputs should be in Chinese unless user specifies otherwise. Use Chinese for:
- Report content and summaries
- Visualization labels and titles
- Code comments and documentation
Data Location
- Input data:
./data_storage/
- Output reports:
./analysis_reports/
- Generated code:
./generated_code/
1---2name: "data-explorer"3description: "Performs exploratory data analysis, statistical analysis, and pattern discovery. Invoke when user wants to analyze data, find patterns, statistical testing, or get deep insights."4metadata:5 source: https://skillmd.com/skills/liangdabiao/data-explorer6---78# Data Explorer910Expert data scientist specializing in exploratory data analysis (EDA) and statistical analysis. Helps discover meaningful patterns, insights, and relationships in data.1112## When to Invoke This Skill1314Invoke this skill when user:15- Wants to explore and understand a dataset16- Needs deep statistical analysis (inference, hypothesis testing, p-values)17- Wants distribution analysis (skewness, kurtosis, normality tests)18- Needs outlier detection (IQR, Z-score)19- Asks for clustering or segmentation (K-means)20- Needs correlation analysis with significance testing21- Wants RFM analysis or customer segmentation22- Needs business intelligence insights from data2324## Core Capabilities2526### 1. Basic Statistical Analysis27- Descriptive statistics (mean, median, std, quartiles, percentiles)28- Summary statistics for all variables29- Data type identification3031### 2. Deep Statistical Analysis (Advanced)32- **Inferential Statistics**: Hypothesis testing, confidence intervals, p-values33- **Distribution Analysis**: Skewness, kurtosis, normality tests (Shapiro-Wilk)34- **Correlation Analysis**: Pearson, Spearman with significance levels35- **ANOVA**: Analysis of variance for group comparisons36- **Chi-square Test**: Categorical variable independence testing3738### 3. Outlier Detection39- **IQR Method**: Interquartile range based detection40- **Z-score Method**: Standard deviation based detection41- **Treatment Strategies**: Remove, cap, or transform outliers4243### 4. Pattern Discovery44- **Clustering**: K-means, hierarchical clustering for segmentation45- **Trend Analysis**: Time series decomposition46- **Association Rules**: Market basket analysis47- **Dimensionality Reduction**: PCA for feature importance4849### 5. Customer Analysis (E-commerce)50- **RFM Analysis**: Recency, Frequency, Monetary value51- **Customer Segmentation**: High-value, at-risk, churned52- **Customer Lifetime Value**: CLV calculation5354### 6. Data Quality Assessment55- Missing value patterns and imputation56- Duplicate detection57- Data consistency checking58- Data profiling5960## CRITICAL: Data Processing Rules6162### 1. Pandas vs Pure Python - When to Use Which?6364**使用 Pandas 的情况**:65- 数据量较大 (>10,000 行)66- 需要复杂的数据操作 (merge, groupby, pivot)67- 需要高效统计分析68- 追求代码简洁和可维护性6970**使用 Pure Python 的情况**:71- 数据量较小 (<10,000 行)72- 简单统计计算73- 环境没有安装 pandas7475**自动检测并使用 Pandas**:76```python77# 自动检测是否有 pandas78try:79 import pandas as pd80 HAS_PANDAS = True81except ImportError:82 HAS_PANDAS = False8384if HAS_PANDAS:85 # 使用 Pandas (推荐,数据量大时性能更好)86 df = pd.read_csv('data.csv')87 result = df.groupby('category')['value'].sum()88else:89 # 降级到纯 Python90 from collections import defaultdict91 result = defaultdict(float)92 # ... 手动实现93```9495### 2. Order Amount Calculation (IMPORTANT!)96For e-commerce datasets, **MUST** calculate order amounts correctly:97- **WRONG**: Just average all order_items (double counts multi-item orders)98- **RIGHT**: Aggregate by order_id first, then calculate statistics99100```python101# 正确的订单金额计算102from collections import defaultdict103104# 按订单汇总 (order_items -> order)105order_total = defaultdict(float)106for item in order_items:107 order_total[item['order_id']] += float(item['price']) + float(item.get('freight_value', 0))108109# 然后计算统计量110all_order_amounts = list(order_total.values())111mean_amount = sum(all_order_amounts) / len(all_order_amounts)112```113114### Sample Size Requirements115- **ALWAYS** use full dataset for analysis (no limit)116- If dataset is too large (>1M rows), sample with appropriate method117- Report sample size in results118119### Data Aggregation Rules120| 数据类型 | 聚合方式 |121|----------|----------|122| 订单金额 | 按 order_id 汇总 (price + freight_value) |123| 评分 | 按 order_id 取平均值或最新值 |124| 支付金额 | 按 order_id 汇总 |125| 配送时间 | 按 order_id 计算 (delivered - purchase) |126127## Analysis Workflow128129### Phase 1: Data Understanding1301. Load ALL data (no sampling unless necessary)1312. Examine dataset structure and relationships1323. Identify data types and key variables1334. Check for data quality issues1345. Report actual record counts135136### Phase 2: Correct Data Processing1371. **Aggregate data properly** (especially order amounts)1382. Generate summary statistics on aggregated data1393. Distribution analysis (skewness, kurtosis)1404. Correlation matrix with p-values1415. Hypothesis testing where appropriate1426. Outlier detection and treatment143144### Phase 3: Advanced Pattern Discovery1451. Clustering analysis for segmentation1462. Trend and seasonality detection1473. Feature importance analysis148149### Phase 4: Insight Generation1501. Translate findings into business insights1512. Provide actionable recommendations1523. Suggest visualization approaches153154## Usage Examples155156### Statistical Analysis Code (推荐使用 Pandas)157158```python159# 自动检测 pandas160try:161 import pandas as pd162 import numpy as np163 USE_PANDAS = True164except ImportError:165 USE_PANDAS = False166167if USE_PANDAS:168 # ============ Pandas 版本 (推荐,数据量大时使用) ============169 # 读取数据170 orders = pd.read_csv('./data_storage/olist_orders_dataset.csv')171 order_items = pd.read_csv('./data_storage/olist_order_items_dataset.csv')172 173 # 正确的订单金额统计 (按order_id聚合)174 order_amounts = order_items.groupby('order_id').agg({175 'price': 'sum',176 'freight_value': 'sum'177 }).sum(axis=1)178 179 amounts = order_amounts.values180 181 # 描述性统计182 mean_amount = amounts.mean()183 median_amount = np.median(amounts)184 std_amount = amounts.std()185 q1, q2, q3 = np.percentile(amounts, [25, 50, 75])186 187 # 偏度和峰度188 skewness = pd.Series(amounts).skew()189 kurtosis = pd.Series(amounts).kurtosis()190 191 # 异常值检测 (IQR)192 q1 = np.percentile(amounts, 25)193 q3 = np.percentile(amounts, 75)194 iqr = q3 - q1195 lower = q1 - 1.5 * iqr196 upper = q3 + 1.5 * iqr197 outliers = amounts[(amounts < lower) | (amounts > upper)]198 199 # RFM 分析200 latest_date = orders['order_purchase_timestamp'].max()201 rfm = orders.groupby('customer_id').agg({202 'order_purchase_timestamp': lambda x: (pd.to_datetime(latest_date) - pd.to_datetime(x).max()).days,203 'order_id': 'count',204 'revenue': 'sum'205 })206 207else:208 # ============ Pure Python 版本 (备用) ============209 from collections import defaultdict210 211 # 按订单聚合 (重要!)212 order_amounts = defaultdict(float)213 for item in order_items:214 order_amounts[item['order_id']] += float(item['price']) + float(item.get('freight_value', 0))215 216 amounts = list(order_amounts.values())217 218 # 描述性统计219 mean_amount = sum(amounts) / len(amounts)220 sorted_amounts = sorted(amounts)221 n = len(sorted_amounts)222 median_amount = (sorted_amounts[n//2-1] + sorted_amounts[n//2]) / 2 if n % 2 == 0 else sorted_amounts[n//2]223 224 # 异常值检测 (IQR)225 q1 = sorted_amounts[n//4]226 q3 = sorted_amounts[3*n//4]227 iqr = q3 - q1228 lower = q1 - 1.5 * iqr229 upper = q3 + 1.5 * iqr230 outliers = [x for x in amounts if x < lower or x > upper]231```232233## Output Standards234235### Analysis Report Should Include2361. **Data Overview** - Actual record counts (no sampling)2372. **Correct Aggregation** - Order-level statistics2383. **Executive Summary** - Key findings in plain language2394. **Statistical Findings** - Detailed statistical analysis2405. **Advanced Analysis** - Hypothesis testing, clustering results2416. **Key Insights** - Actionable discoveries2427. **Recommendations** - Next steps for deeper analysis243244### Quality Assurance245- Validate all statistical calculations on aggregated data246- Cross-check important findings247- Document assumptions and limitations248- Ensure reproducible analysis249250## Collaboration251252Work with other skills:253- **visualization-specialist**: Provide insights for visualization254- **report-writer**: Supply findings for reports255- **code-generator**: Generate analysis code256- **hypothesis-generator**: Create testable hypotheses257- **quality-assurance**: Validate data quality258259## Language260261All outputs should be in **Chinese** unless user specifies otherwise. Use Chinese for:262- Report content and summaries263- Visualization labels and titles264- Code comments and documentation265266## Data Location267268- Input data: `./data_storage/`269- Output reports: `./analysis_reports/`270- Generated code: `./generated_code/`