# Data Ml Pipeline

> Design or review data, analytics, ML, embedding, vector-search, RAG, and model-lifecycle pipelines. Use when the product depends on sourced data, schemas, transformation, training, evaluation, deployment, lineage, drift detection, or retraining. Produce data contracts, reproducibility requirements, evaluation evidence, deployment controls, and monitoring ownership.

- Skill: `ashermahonin/data-ml-pipeline` (Agent Skill, multi-file: 3 files)
- Install (CLI): `npx skillmds@latest add ashermahonin/data-ml-pipeline`
- Raw SKILL.md: https://api.skillmd.com/api/skills/ashermahonin/data-ml-pipeline/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: DevOps & Infra
- Author: ashermahonin (https://skillmd.com/u/ashermahonin)
- Updated: 2026-09-21
- Page: https://skillmd.com/skills/ashermahonin/data-ml-pipeline

---


# Data / ML Pipeline

## Purpose

Define how data is sourced, validated, transformed, versioned, evaluated, deployed, and monitored. Preserve provenance and reproducibility, and do not release a model without representative evaluation and a drift response.

## Product context

1. Read `references/data-ml-stages.md`.
2. Pull architecture context from `architecture-review`, platform matrix from `platform-detector`, security posture from `security-secrets` and `security-owasp-llm` (if LLM-using).
3. Classify the work: analytical warehouse, lakehouse, streaming, feature store, training pipeline, model serving, RAG, fine-tune, eval harness, monitoring.
4. Use Context7 MCP for current docs of: data warehouses (BigQuery, Snowflake, Redshift, ClickHouse), lakehouse (Delta, Iceberg, Hudi), orchestrators (Airflow, Dagster, Prefect), streaming (Kafka, Flink, Materialize), feature stores (Feast, Tecton), ML platforms (MLflow, Weights & Biases, Vertex AI, SageMaker), vector stores (Pinecone, pgvector, Qdrant, Weaviate), and LLM providers.

## Design and validation

1. **Data contracts.** For each dataset: schema, owner, source, freshness SLA, quality SLA, PII fields, retention, access.
2. **Lineage.** Map upstream → transformations → downstream consumers. No orphan datasets.
3. **Quality gates.** Per dataset: schema check, row-count anomaly, null-rate threshold, distribution drift detector, referential integrity.
4. **Versioning.** Datasets and models versioned with reproducible build. Lock training data + code + hyperparameters per artifact.
5. **Evaluation harness.** Per model: train/val/test split policy, evaluation metrics, baseline, fairness audit, robustness probes (adversarial, distribution shift, missing fields).
6. **Deployment.** Canary or shadow deploy; rollback path; monitoring on prediction distribution; A/B test plan if user-facing.
7. **Drift monitoring.** Input drift, output drift, performance drift; retraining triggers.
8. **LLM-specific.** Embedding model pinned; RAG corpus provenance and refresh cadence; eval set per persona; offline + online eval; coordinate with `security-owasp-llm` for injection and disclosure risks.

## Constraints

- Use Context7 MCP for current platform docs, model-card limitations, dataset license terms.
- Keep a decision trace: choice of warehouse/lakehouse/streaming, why; choice of orchestrator; choice of model; rejected options.
- Refuse to deploy a model without an evaluation harness, drift monitor, and retraining trigger.
- Escalate before processing PII, training on user-generated content, or fine-tuning on customer data without explicit consent and a documented retention policy.

## Specification and evidence

- Data flow diagram (sources, transforms, consumers)
- Data contracts (schema, freshness, quality, PII, retention)
- Quality-gate definitions per dataset
- Evaluation harness with baseline metrics
- Deployment plan with rollback and shadow/canary
- Drift-monitoring posture and retraining triggers
- LLM-specific: RAG corpus provenance, eval set, refresh cadence

## Acceptance checks

- No undocumented dataset in production.
- No model deployed without evaluation harness, drift monitor, retraining trigger.
- No PII processed without retention policy and access control.
- No training run without a versioned, reproducible build.
- No LLM-driven product feature without an eval set ≥ 30 representative prompts.

## Handoff

Hand off to `service-implementation` for code, `infrastructure-as-code` for storage/streaming, `cve-zero-day-scanner` for dependency feeds, `security-owasp-llm` / `-agentic` for LLM features, `qa-eval` for eval gates.

## References

- `references/data-ml-stages.md`: pipeline stages, evaluation methods, drift detection, LLM-specific patterns.