# Machine Learning Engineer

> Use when building ML systems, training pipelines, model serving infrastructure, or productionizing ML models at scale. Trigger phrases: "ML pipeline", "model training", "model serving", "inference API", "feature store", "model registry", "training job", "GPU training", "PyTorch model", "TensorFlow serving", "ONNX", "TorchServe", "Triton", "model optimization", "quantization", "distributed training", "ML platform".

- Skill: `barastrong/machine-learning-engineer` (Agent Skill, multi-file: 4 files)
- Install (CLI): `npx skillmds@latest add barastrong/machine-learning-engineer`
- Raw SKILL.md: https://api.skillmd.com/api/skills/barastrong/machine-learning-engineer/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: DevOps & Infra
- Author: barastrong (https://skillmd.com/u/barastrong)
- Updated: 2026-09-22
- Page: https://skillmd.com/skills/barastrong/machine-learning-engineer

---


# Machine Learning Engineer

## Role Summary

A Machine Learning Engineer builds the engineering infrastructure for ML
systems — training pipelines, feature stores, model serving, and monitoring.
The role bridges Data Science (experimentation) and Software Engineering
(production reliability) to deploy ML at scale.

## Core Responsibilities

- Build and maintain ML training pipelines (data → features → train → evaluate)
- Implement feature engineering pipelines and feature stores
- Build model serving infrastructure (REST/gRPC, batch scoring, streaming)
- Optimize models for production (quantization, pruning, distillation, ONNX)
- Implement model monitoring (drift detection, performance degradation)
- Manage ML infrastructure (GPU clusters, distributed training, compute optimization)
- Maintain model registry and experiment tracking
- Collaborate with Data Scientists on productionizing experiments

## Standard Workflow

1. **Experiment Review** — understand model from Data Scientist; assess
   productionization requirements (latency, throughput, scale).
2. **Feature Pipeline** — implement reproducible feature engineering;
   register in feature store.
3. **Training Pipeline** — orchestrated, reproducible training job;
   hyperparameter management.
4. **Model Evaluation** — offline evaluation, A/B shadow mode, champion/challenger.
5. **Serving** — deploy model API (REST/gRPC), batch scoring, or streaming
   inference; load testing.
6. **Monitoring** — prediction drift, feature drift, performance metrics,
   data quality.
7. **Retraining** — automated retraining triggers, pipeline CI/CD.

## Technology Stack

| Layer | Tools |
|-------|-------|
| Training | PyTorch, TensorFlow, JAX, Keras |
| Orchestration | Kubeflow, MLflow, ZenML, Metaflow, Vertex AI Pipelines |
| Feature Store | Feast, Tecton, Hopsworks, Vertex Feature Store |
| Serving | TorchServe, Triton, BentoML, Seldon, Ray Serve, SageMaker |
| Optimization | ONNX, TensorRT, OpenVINO, quantization, pruning |
| Distributed | PyTorch DDP, DeepSpeed, Ray Train, Horovod |
| Monitoring | Evidently, WhyLabs, Arize, Grafana |
| Experiment Tracking | MLflow, Weights & Biases, Neptune |

## Best Practices

- Treat ML code like production code: version control, tests, CI/CD.
- Reproducible training: pin seeds, data versions, dependency versions.
- Decouple feature engineering from model code — reusable feature pipelines.
- Model cards: document training data, intended use, limitations.
- Shadow deployment before traffic cut — validate offline vs. online metrics.
- Latency SLA drives serving architecture: real-time vs. near-real-time vs. batch.
- Monitor input distributions, not just output metrics.
- Canary rollout with automated rollback on metric degradation.

## Anti-Patterns to Avoid

- Training-serving skew: feature engineering differs between train and serve.
- No online monitoring — model degrades silently.
- Single large monolithic training script — hard to test and debug.
- GPU waste: no mixed precision, no gradient checkpointing on large models.
- Deploying without load testing or latency profiling.
- Model without documentation (model card, feature definitions).

## References

- `references/ml-system-design.md` — ML system architecture patterns
- `references/model-serving-guide.md` — serving infrastructure decision guide
- `references/drift-monitoring.md` — drift detection and alerting

## Expected Output Format

1. Training pipeline code (reproducible, versioned)
2. Model artifact + model card (performance metrics, training data summary)
3. Serving API with latency benchmarks
4. Monitoring dashboards + alerting rules