Data Science Engineering Suite - Quick Reference
This skill turns raw data and questions into validated, documented models ready for production:
- EDA workflows: Structured exploration with drift detection
- Feature engineering: Reproducible feature pipelines with leakage prevention and train/serve parity
- Model selection: Baselines first; strong tabular defaults; escalate complexity only when justified
- Evaluation & reporting: Slice analysis, uncertainty, model cards, production metrics
- SQL transformation: SQLMesh for staging/intermediate/marts layers
- MLOps: CI/CD, CT (continuous training), CM (continuous monitoring)
- Production patterns: Data contracts, lineage, feedback loops, streaming features
Modern emphasis (2026): Feature stores, automated retraining, drift monitoring (Evidently), train-serve parity, and agentic ML loops (plan -> execute -> evaluate -> improve). Tools: LightGBM, CatBoost, scikit-learn, PyTorch, Polars (lazy eval for larger-than-RAM datasets), lakeFS for data versioning.
Quick Reference
| Task |
Tool/Framework |
Command |
When to Use |
| EDA & Profiling |
Pandas, Great Expectations |
df.describe(), ge.validate() |
Initial data exploration and quality checks |
| Feature Engineering |
Pandas, Polars, Feature Stores |
df.transform(), Feast materialization |
Creating lag, rolling, categorical features |
| Model Training |
Gradient boosting, linear models, scikit-learn |
lgb.train(), model.fit() |
Strong baselines for tabular ML |
| Hyperparameter Tuning |
Optuna, Ray Tune |
optuna.create_study(), tune.run() |
Optimizing model parameters |
| SQL Transformation |
SQLMesh |
sqlmesh plan, sqlmesh run |
Building staging/intermediate/marts layers |
| Experiment Tracking |
MLflow, W&B |
mlflow.log_metric(), wandb.log() |
Versioning experiments and models |
| Model Evaluation |
scikit-learn, custom metrics |
metrics.roc_auc_score(), slice analysis |
Validating model performance |
Data Lake & Lakehouse
For comprehensive data lake/lakehouse patterns (beyond SQLMesh transformation), see data-lake-platform:
- Table formats: Apache Iceberg, Delta Lake, Apache Hudi
- Query engines: ClickHouse, DuckDB, Apache Doris, StarRocks
- Alternative transformation: dbt (alternative to SQLMesh)
- Ingestion: dlt, Airbyte (connectors)
- Streaming: Apache Kafka patterns
- Orchestration: Dagster, Airflow
This skill focuses on ML feature engineering and modeling. Use data-lake-platform for general-purpose data infrastructure.
Related Skills
For adjacent topics, reference:
- ai-mlops - APIs, batch jobs, monitoring, drift, data ingestion (dlt)
- ai-llm - LLM prompting, fine-tuning, evaluation
- ai-rag - RAG pipelines, chunking, retrieval
- ai-llm-inference - LLM inference optimization, quantization
- ai-ml-timeseries - Time series forecasting, backtesting
- qa-testing-strategy - Test-driven development, coverage
- data-sql-optimization - SQL optimization, index patterns (complements SQLMesh)
- data-lake-platform - Data lake/lakehouse infrastructure (ClickHouse, Iceberg, Kafka)
Decision Tree: Choosing Data Science Approach
User needs ML for: [Problem Type]
- Tabular data?
- Small-medium (<1M rows)? -> LightGBM (fast, efficient)
- Large and complex (>1M rows)? -> LightGBM first, then NN if needed
- High-dim sparse (text, counts)? -> Linear models, then shallow NN
- Time series?
- Seasonality? -> LightGBM, then see ai-ml-timeseries
- Long-term dependencies? -> Transformers (see ai-ml-timeseries)
- Text or mixed modalities?
- LLMs/Transformers -> See ai-llm
- SQL transformations?
- SQLMesh (staging/intermediate/marts layers)
Rule of thumb: For tabular data, tree-based gradient boosting is a strong baseline, but must be validated against alternatives and constraints.
Core Concepts (Vendor-Agnostic)
- Problem framing: define success metrics, baselines, and decision thresholds before modeling.
- Leakage prevention: ensure all features are available at prediction time; split by time/group when appropriate.
- Uncertainty: report confidence intervals and stability (fold variance, bootstrap) rather than single-point metrics.
- Reproducibility: version code/data/features, fix seeds, and record the environment.
- Operational handoff: define monitoring, retraining triggers, and rollback criteria with MLOps.
Implementation Practices (Tooling Examples)
- Track experiments and artifacts (run id, commit hash, data version).
- Add data validation gates in pipelines (schema + distribution + freshness).
- Prefer reproducible, testable feature code (shared transforms, point-in-time correctness).
- Use datasheets/model cards and eval reports as deployment prerequisites (Datasheets for Datasets: https://arxiv.org/abs/1803.09010; Model Cards: https://arxiv.org/abs/1810.03993).
Do / Avoid
Do
- Do start with baselines and a simple model to expose leakage and data issues early.
- Do run slice analysis and document failure modes before recommending deployment.
- Do keep an immutable eval set; refresh training data without contaminating evaluation.
Avoid
- Avoid random splits for temporal or user-correlated data.
- Avoid "metric gaming" (optimizing the number without validating business impact).
- Avoid training on labels created after the prediction timestamp (silent future leakage).
Core Patterns (Overview)
Pattern 1: End-to-End DS Project Lifecycle
Use when: Starting or restructuring any DS/ML project.
Stages:
- Problem framing - Business objective, success metrics, baseline
- Data & feasibility - Sources, coverage, granularity, label quality
- EDA & data quality - Schema, missingness, outliers, leakage checks
- Feature engineering - Per data type with feature store integration
- Modelling - Baselines first, then LightGBM, then complexity as needed
- Evaluation - Offline metrics, slice analysis, error analysis
- Reporting - Model evaluation report + model card
- MLOps - CI/CD, CT (continuous training), CM (continuous monitoring)
Detailed guide: EDA Best Practices
Pattern 2: Feature Engineering
Use when: Designing features before modelling or during model improvement.
By data type:
- Numeric: Standardize, handle outliers, transform skew, scale
- Categorical: One-hot/ordinal (low cardinality), target/frequency/hashing (high cardinality)
- Feature Store Integration: Store encoders, mappings, statistics centrally
- Text: Cleaning, TF-IDF, embeddings, simple stats
- Time: Calendar features, recency, rolling/lag features
Key Modern Practice: Use feature stores (Feast, Tecton, Databricks) for versioning, sharing, and train-serve parity.
Detailed guide: Feature Engineering Patterns
Pattern 3: Data Contracts & Lineage
Use when: Building production ML systems with data quality requirements.
Components:
- Contracts: Schema + ranges/nullability + freshness SLAs
- Lineage: Track source -> feature store -> train -> serve
- Feature store hygiene: Materialization cadence, backfill/replay, encoder versioning
- Schema evolution: Backward/forward-compatible migrations with shadow runs
Detailed guide: Data Contracts & Lineage
Pattern 4: Model Selection & Training
Use when: Picking model families and starting experiments.
Decision guide (modern benchmarks):
- Tabular: Start with a strong baseline (linear/logistic, then gradient boosting) and iterate based on error analysis
- Baselines: Always implement simple baselines first (majority class, mean, naive forecast)
- Train/val/test splits: Time-based (forecasting), group-based (user/item leakage), or random (IID)
- Hyperparameter tuning: Start manual, then Bayesian optimization (Optuna, Ray Tune)
- Overfitting control: Regularization, early stopping, cross-validation
Detailed guide: Modelling Patterns
Pattern 5: Evaluation & Reporting
Use when: Finalizing a model candidate or handing over to production.
Key components:
- Metric selection: Primary (ROC-AUC, PR-AUC, RMSE) + guardrails (calibration, fairness)
- Threshold selection: ROC/PR curves, cost-sensitive, F1 maximization
- Slice analysis: Performance by geography, user segments, product categories
- Error analysis: Collect high-error examples, cluster by error type, identify systematic failures
- Uncertainty: Confidence intervals (bootstrap where appropriate), variance across folds, and stability checks
- Evaluation report: 8-section report (objective, data, features, models, metrics, slices, risks, recommendation)
- Model card: Documentation for stakeholders (intended use, data, performance, ethics, operations)
Detailed guide: Evaluation Patterns
Pattern 6: Reproducibility & MLOps
Use when: Ensuring experiments are reproducible and production-ready.
Modern MLOps (CI/CD/CT/CM):
- CI (Continuous Integration): Automated testing, data validation, code quality
- CD (Continuous Delivery): Environment-specific promotion (dev -> staging -> prod), canary deployment
- CT (Continuous Training): Drift-triggered and scheduled retraining
- CM (Continuous Monitoring): Real-time data drift, performance, system health
Versioning:
- Code (git commit), data (DVC, LakeFS), features (feature store), models (MLflow Registry)
- Seeds (reproducibility), hyperparameters (experiment tracker)
Detailed guide: Reproducibility Checklist
Pattern 7: Feature Freshness & Streaming
Use when: Managing real-time features and streaming pipelines.
Components:
- Freshness contracts: Define freshness SLAs per feature, monitor lag, alert on breaches
- Batch + stream parity: Same feature logic across batch/stream, idempotent upserts
- Schema evolution: Version schemas, add forward/backward-compatible parsers, backfill with rollback
- Data quality gates: PII/format checks, range checks, distribution drift (KL, KS, PSI)
Detailed guide: Feature Freshness & Streaming
Pattern 8: Production Feedback Loops
Use when: Capturing production signals and implementing continuous improvement.
Components:
- Signal capture: Log predictions + user edits/acceptance/abandonment (scrub PII)
- Labeling: Route failures/edge cases to human review, create balanced sets
- Dataset refresh: Periodic refresh (weekly/monthly) with lineage, protect eval set
- Online eval: Shadow/canary new models, track solve rate, calibration, cost, latency
Detailed guide: Production Feedback Loops
Resources (Detailed Guides)
For comprehensive operational patterns and checklists, see:
- EDA Best Practices - Structured workflow for exploratory data analysis
- Feature Engineering Patterns - Operational patterns by data type
- Data Contracts & Lineage - Data quality, versioning, feature store ops
- Modelling Patterns - Model selection, hyperparameter tuning, train/test splits
- Evaluation Patterns - Metrics, slice analysis, evaluation reports, model cards
- Reproducibility Checklist - Experiment tracking, MLOps (CI/CD/CT/CM)
- Feature Freshness & Streaming - Real-time features, schema evolution
- Production Feedback Loops - Online learning, labeling, canary deployment
Templates
Use these as copy-paste starting points:
Project & Workflow Templates
- Standard DS project template:
assets/project/template-standard.md
- Quick DS experiment template:
assets/project/template-quick.md
Feature Engineering & EDA
- Feature engineering template:
assets/features/template-feature-engineering.md
- EDA checklist & notebook template:
assets/eda/template-eda.md
Evaluation & Reporting
- Model evaluation report:
assets/evaluation/template-evaluation-report.md
- Model card:
assets/evaluation/template-model-card.md
- ML experiment review:
assets/review/experiment-review-template.md
SQL Transformation (SQLMesh)
For SQL-based data transformation and feature engineering:
- SQLMesh project setup:
../data-lake-platform/assets/transformation/sqlmesh/template-sqlmesh-project.md
- SQLMesh model types:
../data-lake-platform/assets/transformation/sqlmesh/template-sqlmesh-model.md (FULL, INCREMENTAL, VIEW)
- Incremental models:
../data-lake-platform/assets/transformation/sqlmesh/template-sqlmesh-incremental.md
- DAG and dependencies:
../data-lake-platform/assets/transformation/sqlmesh/template-sqlmesh-dag.md
- Testing and data quality:
../data-lake-platform/assets/transformation/sqlmesh/template-sqlmesh-testing.md
Use SQLMesh when:
- Building SQL-based feature pipelines
- Managing incremental data transformations
- Creating staging/intermediate/marts layers
- Testing SQL logic with unit tests and audits
For data ingestion (loading raw data), use:
- ai-mlops skill (dlt templates for REST APIs, databases, warehouses)
Navigation
Resources
- references/reproducibility-checklist.md
- references/evaluation-patterns.md
- references/feature-engineering-patterns.md
- references/modelling-patterns.md
- references/feature-freshness-streaming.md
- references/eda-best-practices.md
- references/data-contracts-lineage.md
- references/production-feedback-loops.md
Templates
Data
- data/sources.json - Curated external references
External Resources
See data/sources.json for curated foundational and implementation references:
- Core ML/DL: scikit-learn, XGBoost, LightGBM, PyTorch, TensorFlow, JAX
- Data processing: pandas, NumPy, Polars, DuckDB, Spark, Dask
- SQL transformation: SQLMesh, dbt (staging/marts/incremental patterns)
- Feature stores: Feast, Tecton, Databricks Feature Store (centralized feature management)
- Data validation: Pydantic, Great Expectations, Pandera, Evidently (quality + drift)
- Visualization: Matplotlib, Seaborn, Plotly, Streamlit, Dash
- MLOps: MLflow, W&B, DVC, Neptune (experiment tracking + model registry)
- Hyperparameter tuning: Optuna, Ray Tune, Hyperopt
- Model serving: BentoML, FastAPI, TorchServe, Seldon, Ray Serve
- Orchestration: Kubeflow, Metaflow, Prefect, Airflow, ZenML
- Cloud platforms: AWS SageMaker, Google Vertex AI, Azure ML, Databricks, Snowflake
Use this skill to execute data science projects end-to-end: concrete checklists, patterns, and templates, not theory.
1---2name: ai-ml-data-science3description: "End-to-end data science and ML engineering workflows: problem framing, data/EDA, feature engineering (feature stores), modelling, evaluation/reporting, plus SQL transformations with SQLMesh. Use for dataset exploration, feature design, model selection, metrics and slice analysis, model cards/eval reports, experiment reproducibility, and production handoff (monitoring and retraining)."4metadata:5 source: https://skillmd.com/skills/diegosouzapw/ai-ml-data-science6---78# Data Science Engineering Suite - Quick Reference910This skill turns **raw data and questions** into **validated, documented models** ready for production:1112- **EDA workflows**: Structured exploration with drift detection13- **Feature engineering**: Reproducible feature pipelines with leakage prevention and train/serve parity14- **Model selection**: Baselines first; strong tabular defaults; escalate complexity only when justified15- **Evaluation & reporting**: Slice analysis, uncertainty, model cards, production metrics16- **SQL transformation**: SQLMesh for staging/intermediate/marts layers17- **MLOps**: CI/CD, CT (continuous training), CM (continuous monitoring)18- **Production patterns**: Data contracts, lineage, feedback loops, streaming features1920**Modern emphasis (2026):** Feature stores, automated retraining, drift monitoring (Evidently), train-serve parity, and agentic ML loops (plan -> execute -> evaluate -> improve). Tools: LightGBM, CatBoost, scikit-learn, PyTorch, Polars (lazy eval for larger-than-RAM datasets), lakeFS for data versioning.2122---2324## Quick Reference2526| Task | Tool/Framework | Command | When to Use |27|------|----------------|---------|-------------|28| EDA & Profiling | Pandas, Great Expectations | `df.describe()`, `ge.validate()` | Initial data exploration and quality checks |29| Feature Engineering | Pandas, Polars, Feature Stores | `df.transform()`, Feast materialization | Creating lag, rolling, categorical features |30| Model Training | Gradient boosting, linear models, scikit-learn | `lgb.train()`, `model.fit()` | Strong baselines for tabular ML |31| Hyperparameter Tuning | Optuna, Ray Tune | `optuna.create_study()`, `tune.run()` | Optimizing model parameters |32| SQL Transformation | SQLMesh | `sqlmesh plan`, `sqlmesh run` | Building staging/intermediate/marts layers |33| Experiment Tracking | MLflow, W&B | `mlflow.log_metric()`, `wandb.log()` | Versioning experiments and models |34| Model Evaluation | scikit-learn, custom metrics | `metrics.roc_auc_score()`, slice analysis | Validating model performance |3536---3738## Data Lake & Lakehouse3940For comprehensive data lake/lakehouse patterns (beyond SQLMesh transformation), see **[data-lake-platform](../data-lake-platform/SKILL.md)**:4142- **Table formats:** Apache Iceberg, Delta Lake, Apache Hudi43- **Query engines:** ClickHouse, DuckDB, Apache Doris, StarRocks44- **Alternative transformation:** dbt (alternative to SQLMesh)45- **Ingestion:** dlt, Airbyte (connectors)46- **Streaming:** Apache Kafka patterns47- **Orchestration:** Dagster, Airflow4849This skill focuses on **ML feature engineering and modeling**. Use data-lake-platform for general-purpose data infrastructure.5051---5253## Related Skills5455For adjacent topics, reference:5657- **[ai-mlops](../ai-mlops/SKILL.md)** - APIs, batch jobs, monitoring, drift, data ingestion (dlt)58- **[ai-llm](../ai-llm/SKILL.md)** - LLM prompting, fine-tuning, evaluation59- **[ai-rag](../ai-rag/SKILL.md)** - RAG pipelines, chunking, retrieval60- **[ai-llm-inference](../ai-llm-inference/SKILL.md)** - LLM inference optimization, quantization61- **[ai-ml-timeseries](../ai-ml-timeseries/SKILL.md)** - Time series forecasting, backtesting62- **[qa-testing-strategy](../qa-testing-strategy/SKILL.md)** - Test-driven development, coverage63- **[data-sql-optimization](../data-sql-optimization/SKILL.md)** - SQL optimization, index patterns (complements SQLMesh)64- **[data-lake-platform](../data-lake-platform/SKILL.md)** - Data lake/lakehouse infrastructure (ClickHouse, Iceberg, Kafka)6566---6768## Decision Tree: Choosing Data Science Approach6970```text71User needs ML for: [Problem Type]72 - Tabular data?73 - Small-medium (<1M rows)? -> LightGBM (fast, efficient)74 - Large and complex (>1M rows)? -> LightGBM first, then NN if needed75 - High-dim sparse (text, counts)? -> Linear models, then shallow NN7677 - Time series?78 - Seasonality? -> LightGBM, then see ai-ml-timeseries79 - Long-term dependencies? -> Transformers (see ai-ml-timeseries)8081 - Text or mixed modalities?82 - LLMs/Transformers -> See ai-llm8384 - SQL transformations?85 - SQLMesh (staging/intermediate/marts layers)86```8788**Rule of thumb:** For tabular data, tree-based gradient boosting is a strong baseline, but must be validated against alternatives and constraints.8990---9192## Core Concepts (Vendor-Agnostic)9394- **Problem framing**: define success metrics, baselines, and decision thresholds before modeling.95- **Leakage prevention**: ensure all features are available at prediction time; split by time/group when appropriate.96- **Uncertainty**: report confidence intervals and stability (fold variance, bootstrap) rather than single-point metrics.97- **Reproducibility**: version code/data/features, fix seeds, and record the environment.98- **Operational handoff**: define monitoring, retraining triggers, and rollback criteria with MLOps.99100## Implementation Practices (Tooling Examples)101102- Track experiments and artifacts (run id, commit hash, data version).103- Add data validation gates in pipelines (schema + distribution + freshness).104- Prefer reproducible, testable feature code (shared transforms, point-in-time correctness).105- Use datasheets/model cards and eval reports as deployment prerequisites (Datasheets for Datasets: https://arxiv.org/abs/1803.09010; Model Cards: https://arxiv.org/abs/1810.03993).106107## Do / Avoid108109**Do**110- Do start with baselines and a simple model to expose leakage and data issues early.111- Do run slice analysis and document failure modes before recommending deployment.112- Do keep an immutable eval set; refresh training data without contaminating evaluation.113114**Avoid**115- Avoid random splits for temporal or user-correlated data.116- Avoid "metric gaming" (optimizing the number without validating business impact).117- Avoid training on labels created after the prediction timestamp (silent future leakage).118119# Core Patterns (Overview)120121## Pattern 1: End-to-End DS Project Lifecycle122123**Use when:** Starting or restructuring any DS/ML project.124125**Stages:**1261271. **Problem framing** - Business objective, success metrics, baseline1282. **Data & feasibility** - Sources, coverage, granularity, label quality1293. **EDA & data quality** - Schema, missingness, outliers, leakage checks1304. **Feature engineering** - Per data type with feature store integration1315. **Modelling** - Baselines first, then LightGBM, then complexity as needed1326. **Evaluation** - Offline metrics, slice analysis, error analysis1337. **Reporting** - Model evaluation report + model card1348. **MLOps** - CI/CD, CT (continuous training), CM (continuous monitoring)135136**Detailed guide:** [EDA Best Practices](references/eda-best-practices.md)137138---139140## Pattern 2: Feature Engineering141142**Use when:** Designing features before modelling or during model improvement.143144**By data type:**145146- **Numeric:** Standardize, handle outliers, transform skew, scale147- **Categorical:** One-hot/ordinal (low cardinality), target/frequency/hashing (high cardinality)148 - **Feature Store Integration:** Store encoders, mappings, statistics centrally149- **Text:** Cleaning, TF-IDF, embeddings, simple stats150- **Time:** Calendar features, recency, rolling/lag features151152**Key Modern Practice:** Use feature stores (Feast, Tecton, Databricks) for versioning, sharing, and train-serve parity.153154**Detailed guide:** [Feature Engineering Patterns](references/feature-engineering-patterns.md)155156---157158## Pattern 3: Data Contracts & Lineage159160**Use when:** Building production ML systems with data quality requirements.161162**Components:**163164- **Contracts:** Schema + ranges/nullability + freshness SLAs165- **Lineage:** Track source -> feature store -> train -> serve166- **Feature store hygiene:** Materialization cadence, backfill/replay, encoder versioning167- **Schema evolution:** Backward/forward-compatible migrations with shadow runs168169**Detailed guide:** [Data Contracts & Lineage](references/data-contracts-lineage.md)170171---172173## Pattern 4: Model Selection & Training174175**Use when:** Picking model families and starting experiments.176177**Decision guide (modern benchmarks):**178179- **Tabular:** Start with a **strong baseline** (linear/logistic, then gradient boosting) and iterate based on error analysis180- **Baselines:** Always implement simple baselines first (majority class, mean, naive forecast)181- **Train/val/test splits:** Time-based (forecasting), group-based (user/item leakage), or random (IID)182- **Hyperparameter tuning:** Start manual, then Bayesian optimization (Optuna, Ray Tune)183- **Overfitting control:** Regularization, early stopping, cross-validation184185**Detailed guide:** [Modelling Patterns](references/modelling-patterns.md)186187---188189## Pattern 5: Evaluation & Reporting190191**Use when:** Finalizing a model candidate or handing over to production.192193**Key components:**194195- **Metric selection:** Primary (ROC-AUC, PR-AUC, RMSE) + guardrails (calibration, fairness)196- **Threshold selection:** ROC/PR curves, cost-sensitive, F1 maximization197- **Slice analysis:** Performance by geography, user segments, product categories198- **Error analysis:** Collect high-error examples, cluster by error type, identify systematic failures199- **Uncertainty:** Confidence intervals (bootstrap where appropriate), variance across folds, and stability checks200- **Evaluation report:** 8-section report (objective, data, features, models, metrics, slices, risks, recommendation)201- **Model card:** Documentation for stakeholders (intended use, data, performance, ethics, operations)202203**Detailed guide:** [Evaluation Patterns](references/evaluation-patterns.md)204205---206207## Pattern 6: Reproducibility & MLOps208209**Use when:** Ensuring experiments are reproducible and production-ready.210211**Modern MLOps (CI/CD/CT/CM):**212213- **CI (Continuous Integration):** Automated testing, data validation, code quality214- **CD (Continuous Delivery):** Environment-specific promotion (dev -> staging -> prod), canary deployment215- **CT (Continuous Training):** Drift-triggered and scheduled retraining216- **CM (Continuous Monitoring):** Real-time data drift, performance, system health217218**Versioning:**219- Code (git commit), data (DVC, LakeFS), features (feature store), models (MLflow Registry)220- Seeds (reproducibility), hyperparameters (experiment tracker)221222**Detailed guide:** [Reproducibility Checklist](references/reproducibility-checklist.md)223224---225226## Pattern 7: Feature Freshness & Streaming227228**Use when:** Managing real-time features and streaming pipelines.229230**Components:**231232- **Freshness contracts:** Define freshness SLAs per feature, monitor lag, alert on breaches233- **Batch + stream parity:** Same feature logic across batch/stream, idempotent upserts234- **Schema evolution:** Version schemas, add forward/backward-compatible parsers, backfill with rollback235- **Data quality gates:** PII/format checks, range checks, distribution drift (KL, KS, PSI)236237**Detailed guide:** [Feature Freshness & Streaming](references/feature-freshness-streaming.md)238239---240241## Pattern 8: Production Feedback Loops242243**Use when:** Capturing production signals and implementing continuous improvement.244245**Components:**246247- **Signal capture:** Log predictions + user edits/acceptance/abandonment (scrub PII)248- **Labeling:** Route failures/edge cases to human review, create balanced sets249- **Dataset refresh:** Periodic refresh (weekly/monthly) with lineage, protect eval set250- **Online eval:** Shadow/canary new models, track solve rate, calibration, cost, latency251252**Detailed guide:** [Production Feedback Loops](references/production-feedback-loops.md)253254---255256## Resources (Detailed Guides)257258For comprehensive operational patterns and checklists, see:259260- [EDA Best Practices](references/eda-best-practices.md) - Structured workflow for exploratory data analysis261- [Feature Engineering Patterns](references/feature-engineering-patterns.md) - Operational patterns by data type262- [Data Contracts & Lineage](references/data-contracts-lineage.md) - Data quality, versioning, feature store ops263- [Modelling Patterns](references/modelling-patterns.md) - Model selection, hyperparameter tuning, train/test splits264- [Evaluation Patterns](references/evaluation-patterns.md) - Metrics, slice analysis, evaluation reports, model cards265- [Reproducibility Checklist](references/reproducibility-checklist.md) - Experiment tracking, MLOps (CI/CD/CT/CM)266- [Feature Freshness & Streaming](references/feature-freshness-streaming.md) - Real-time features, schema evolution267- [Production Feedback Loops](references/production-feedback-loops.md) - Online learning, labeling, canary deployment268269---270271## Templates272273Use these as copy-paste starting points:274275### Project & Workflow Templates276277- **Standard DS project template:** `assets/project/template-standard.md`278- **Quick DS experiment template:** `assets/project/template-quick.md`279280### Feature Engineering & EDA281282- **Feature engineering template:** `assets/features/template-feature-engineering.md`283- **EDA checklist & notebook template:** `assets/eda/template-eda.md`284285### Evaluation & Reporting286287- **Model evaluation report:** `assets/evaluation/template-evaluation-report.md`288- **Model card:** `assets/evaluation/template-model-card.md`289- **ML experiment review:** `assets/review/experiment-review-template.md`290291### SQL Transformation (SQLMesh)292293For SQL-based data transformation and feature engineering:294295- **SQLMesh project setup:** `../data-lake-platform/assets/transformation/sqlmesh/template-sqlmesh-project.md`296- **SQLMesh model types:** `../data-lake-platform/assets/transformation/sqlmesh/template-sqlmesh-model.md` (FULL, INCREMENTAL, VIEW)297- **Incremental models:** `../data-lake-platform/assets/transformation/sqlmesh/template-sqlmesh-incremental.md`298- **DAG and dependencies:** `../data-lake-platform/assets/transformation/sqlmesh/template-sqlmesh-dag.md`299- **Testing and data quality:** `../data-lake-platform/assets/transformation/sqlmesh/template-sqlmesh-testing.md`300301**Use SQLMesh when:**302- Building SQL-based feature pipelines303- Managing incremental data transformations304- Creating staging/intermediate/marts layers305- Testing SQL logic with unit tests and audits306307**For data ingestion (loading raw data), use:**308- [ai-mlops](../ai-mlops/SKILL.md) skill (dlt templates for REST APIs, databases, warehouses)309310## Navigation311312**Resources**313- [references/reproducibility-checklist.md](references/reproducibility-checklist.md)314- [references/evaluation-patterns.md](references/evaluation-patterns.md)315- [references/feature-engineering-patterns.md](references/feature-engineering-patterns.md)316- [references/modelling-patterns.md](references/modelling-patterns.md)317- [references/feature-freshness-streaming.md](references/feature-freshness-streaming.md)318- [references/eda-best-practices.md](references/eda-best-practices.md)319- [references/data-contracts-lineage.md](references/data-contracts-lineage.md)320- [references/production-feedback-loops.md](references/production-feedback-loops.md)321322**Templates**323- [assets/project/template-standard.md](assets/project/template-standard.md)324- [assets/project/template-quick.md](assets/project/template-quick.md)325- [assets/features/template-feature-engineering.md](assets/features/template-feature-engineering.md)326- [assets/eda/template-eda.md](assets/eda/template-eda.md)327- [assets/evaluation/template-evaluation-report.md](assets/evaluation/template-evaluation-report.md)328- [assets/evaluation/template-model-card.md](assets/evaluation/template-model-card.md)329- [assets/review/experiment-review-template.md](assets/review/experiment-review-template.md)330- [template-sqlmesh-project.md](../data-lake-platform/assets/transformation/sqlmesh/template-sqlmesh-project.md)331- [template-sqlmesh-model.md](../data-lake-platform/assets/transformation/sqlmesh/template-sqlmesh-model.md)332- [template-sqlmesh-incremental.md](../data-lake-platform/assets/transformation/sqlmesh/template-sqlmesh-incremental.md)333- [template-sqlmesh-dag.md](../data-lake-platform/assets/transformation/sqlmesh/template-sqlmesh-dag.md)334- [template-sqlmesh-testing.md](../data-lake-platform/assets/transformation/sqlmesh/template-sqlmesh-testing.md)335336**Data**337- [data/sources.json](data/sources.json) - Curated external references338339---340341## External Resources342343See [data/sources.json](data/sources.json) for curated foundational and implementation references:344345- **Core ML/DL**: scikit-learn, XGBoost, LightGBM, PyTorch, TensorFlow, JAX346- **Data processing**: pandas, NumPy, Polars, DuckDB, Spark, Dask347- **SQL transformation**: SQLMesh, dbt (staging/marts/incremental patterns)348- **Feature stores**: Feast, Tecton, Databricks Feature Store (centralized feature management)349- **Data validation**: Pydantic, Great Expectations, Pandera, Evidently (quality + drift)350- **Visualization**: Matplotlib, Seaborn, Plotly, Streamlit, Dash351- **MLOps**: MLflow, W&B, DVC, Neptune (experiment tracking + model registry)352- **Hyperparameter tuning**: Optuna, Ray Tune, Hyperopt353- **Model serving**: BentoML, FastAPI, TorchServe, Seldon, Ray Serve354- **Orchestration**: Kubeflow, Metaflow, Prefect, Airflow, ZenML355- **Cloud platforms**: AWS SageMaker, Google Vertex AI, Azure ML, Databricks, Snowflake356357Use this skill to **execute data science projects end-to-end**: concrete checklists, patterns, and templates, not theory.