# Bigdata Machine Learning

> Machine learning toolkit for big data teams. Includes scikit-learn, PyTorch Lightning, Transformers, SHAP for model training, deployment, and interpretation. Use when building ML pipelines, training models, or explaining predictions. Use when this capability is needed.

- Skill: `tomevault-io/bigdata-machine-learning` (Agent Skill, multi-file: 2 files)
- Install (CLI): `npx skillmds@latest add tomevault-io/bigdata-machine-learning`
- Raw SKILL.md: https://api.skillmd.com/api/skills/tomevault-io/bigdata-machine-learning/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: tomevault-io (https://skillmd.com/u/tomevault-io)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/tomevault-io/bigdata-machine-learning

---


## 是什么

帮你把"凭直觉判断"的业务环节，变成"用历史数据训练出来的可解释模型"，让流失预警、风险评分、需求预测这种事不再靠 PM 拍脑袋。配合 SHAP（特征贡献度解释，告诉你模型为什么这么判）让模型结果可向业务方解释，不再是黑盒。

## 怎么用

1. 先把业务问题翻译成"输入特征 → 预测目标"的样子，比如"过去 30 天行为 → 下月是否流失"，目标含糊就先不要训。
2. 用 scikit-learn（传统机器学习库，覆盖 90% 业务场景）先跑一个最简单的基线模型，看预测准度有没有比拍脑袋强。
3. 数据量大或要做深度学习（图像/文本/复杂序列）再上 PyTorch Lightning（深度学习训练框架，把样板代码消掉）。
4. 模型上线前必跑 SHAP 解释，挑 3–5 个特征贡献度最高的因子让业务方过目，确认没有反常识的逻辑。
5. 留好"模型版本-训练数据-评估指标"三件套留档，下次模型衰减时能立刻定位是数据漂移还是逻辑过期。

## 架构图

```mermaid
flowchart LR
    A[业务问题] --> B[特征工程]
    B --> C[scikit-learn<br/>基线模型]
    C --> D{效果够?}
    D -->|否| E[PyTorch Lightning<br/>深度学习]
    D -->|是| F[SHAP 解释]
    E --> F
    F --> G[业务方过目]
```

# Big Data Machine Learning Toolkit

## Overview

大数据团队机器学习工具集，从传统ML到深度学习全覆盖。

## Quick Reference

| 工具 | 场景 | 规模 |
|------|------|------|
| **scikit-learn** | 传统ML | 中等数据 |
| **PyTorch Lightning** | 深度学习 | GPU训练 |
| **Transformers** | NLP/LLM | 预训练模型 |
| **SHAP** | 模型解释 | 可解释AI |

## 选择指南

```
任务类型:
├── 分类/回归 → scikit-learn
├── 时间序列 → scikit-learn + statsmodels
├── 文本处理 → Transformers
├── 图像处理 → PyTorch Lightning
├── 强化学习 → stable-baselines3
└── 模型解释 → SHAP
```

## 子Skills

- `scikit-learn/` - 传统机器学习
- `pytorch-lightning/` - 深度学习框架
- `transformers/` - NLP预训练模型
- `shap/` - 模型可解释性
- `stable-baselines3/` - 强化学习

## 常用模式

### 标准ML Pipeline (scikit-learn)
```python
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import cross_val_score

pipeline = Pipeline([
    ('scaler', StandardScaler()),
    ('classifier', RandomForestClassifier())
])

scores = cross_val_score(pipeline, X, y, cv=5)
print(f"CV Score: {scores.mean():.3f} ± {scores.std():.3f}")
```

### 深度学习训练 (PyTorch Lightning)
```python
import pytorch_lightning as pl
from pytorch_lightning.callbacks import EarlyStopping

trainer = pl.Trainer(
    max_epochs=100,
    accelerator="gpu",
    callbacks=[EarlyStopping(monitor="val_loss")]
)
trainer.fit(model, train_loader, val_loader)
```

### NLP任务 (Transformers)
```python
from transformers import pipeline

# 文本分类
classifier = pipeline("text-classification", model="bert-base-chinese")
result = classifier("这个产品质量很好")

# 文本生成
generator = pipeline("text-generation", model="gpt2")
```

### 模型解释 (SHAP)
```python
import shap

explainer = shap.TreeExplainer(model)
shap_values = explainer.shap_values(X)

# 特征重要性
shap.summary_plot(shap_values, X)

# 单样本解释
shap.force_plot(explainer.expected_value, shap_values[0], X.iloc[0])
```

## 大数据ML最佳实践

### 1. 大规模训练
```python
# 使用Dask-ML
from dask_ml.model_selection import train_test_split
from dask_ml.linear_model import LogisticRegression

X_train, X_test = train_test_split(X_dask, y_dask)
model = LogisticRegression()
model.fit(X_train, y_train)
```

### 2. 增量学习
```python
from sklearn.linear_model import SGDClassifier

model = SGDClassifier()
for chunk in data_chunks:
    model.partial_fit(chunk.X, chunk.y, classes=[0, 1])
```

### 3. 模型版本管理
```python
import mlflow

with mlflow.start_run():
    mlflow.log_params(params)
    mlflow.log_metrics(metrics)
    mlflow.sklearn.log_model(model, "model")
```

## 团队规范

1. **实验追踪**: 使用MLflow记录所有实验
2. **模型版本**: 模型+数据版本绑定
3. **可解释性**: 关键模型必须提供SHAP解释
4. **部署流程**: 模型 → 测试 → 审核 → 上线

---

猪哥云-数据产品部 | 大数据团队专用

---
> Source: [MARUCIE/openclaw-foundry](https://github.com/MARUCIE/openclaw-foundry) — distributed by [TomeVault](https://tomevault.io).
<!-- tomevault:4.0:skill_md:2026-05-22 -->