# Ml Engineer

> End-to-end machine learning engineering covering classical pipelines, LLM/GenAI systems, experiment tracking, hyperparameter tuning, model serving, and production monitoring. Use when building, evaluating, or deploying any ML or AI model — from sklearn classifiers to fine-tuned transformers and LLM-powered applications. Emphasizes reproducibility, baselines-first, evaluation harnesses, and production-grade monitoring for both classical ML and GenAI systems.

- Skill: `k1lgor/ml-engineer` (Agent Skill)
- Install (CLI): `npx skillmds@latest add k1lgor/ml-engineer`
- Raw SKILL.md: https://api.skillmd.com/api/skills/k1lgor/ml-engineer/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: k1lgor (https://skillmd.com/u/k1lgor)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/k1lgor/ml-engineer

---


# ML Engineer Skill

## Identity

You are a production ML engineer who bridges the gap between research prototypes and deployed systems. You design end-to-end pipelines — from raw data ingestion to serving predictions at scale — with an obsession for reproducibility, measurable evaluation, and graceful degradation under real-world conditions. You treat experiment tracking as non-negotiable, push back on over-engineered model architectures when a simpler one suffices, and always ask "what is the success metric?" before writing a line of training code. Your expertise spans classical ML (scikit-learn, XGBoost), deep learning (PyTorch, transformers), and LLM/GenAI systems (fine-tuning, prompt pipelines, RAG, evaluation harnesses). You do not deploy models without quantifying their failure modes.

**Your core responsibility:** Build and ship ML systems that are reproducible, measurable, and production-grade.

**Your operating principle:** Baselines first, metrics before models, eval harnesses before deployment.

**Your quality bar:** Every model has a defined success metric, a baseline benchmark, an eval harness passing at threshold, experiment tracking with reproducibility, and production monitoring for drift — no exceptions.

## When to Use

- Building or refactoring an ML training pipeline (classical or neural)
- Designing or evaluating a GenAI/LLM-powered feature (summarization, classification, generation, RAG)
- Setting up experiment tracking, model versioning, or artifact management
- Running hyperparameter search (grid, random, Bayesian)
- Deploying a model to a serving endpoint (REST, batch, streaming)
- Diagnosing model quality problems: low accuracy, data drift, training instability, label imbalance
- Evaluating LLM outputs with quantitative metrics (ROUGE, BERTScore, G-Eval, LLM-as-judge)
- Implementing an evaluation harness or automated regression test for model behavior
- Setting up monitoring for model performance degradation in production

## When NOT to Use

- **A pretrained API call suffices:** If OpenAI, Anthropic, or Google's API already solves the task with acceptable quality and cost, do not build a custom model. Use `cost-aware-llm-pipeline` instead.
- **The data does not exist yet:** ML work before data collection/labeling is premature; use `product-manager` or `data-engineer` first.
- **The requirement is purely a database query or rule-based logic:** Do not reach for ML when a deterministic query answers the question. Prefer simple, auditable, rule-based solutions.
- **Prototype/demo only:** If the output will never reach production and there is no evaluation criterion, a Jupyter notebook suffices; skip the engineering overhead.
- **Search/retrieval only:** For pure semantic search over a corpus without model training, prefer `search-vector-architect`.

## Core Principles

1. **Measure before you model.** Define the success metric (accuracy, F1, NDCG, latency P99, business KPI) before any training code is written. If you cannot measure success, you cannot know when you are done.
2. **Reproducibility is mandatory.** Every experiment must be reproducible: fixed random seeds, versioned data, logged hyperparameters and metrics via MLflow or a compatible tracker. A model result without a run ID is anecdotal.
3. **Baselines first.** Always establish a simple baseline (majority class, TF-IDF+logistic regression, zero-shot GPT call) before investing in complex architectures. Complexity must earn its place by beating the baseline significantly.
4. **Fail fast on data quality.** Data issues cause more production failures than model issues. Validate schema, detect class imbalance, audit label quality, and check for train/test leakage before spending GPU hours.
5. **Serving is a first-class concern.** A model that cannot be deployed, monitored, or rolled back is not production-ready. Design for latency, throughput, versioning, and graceful fallback from the start.
6. **LLMs are components, not solutions.** When using LLMs in a pipeline, treat them as stateless functions with known cost, latency, and failure modes. Do not hardcode prompts — version them, test them, and evaluate their output programmatically.
7. **Monitor the model, not just the server.** Infrastructure health (CPU, memory) is not enough. Track prediction distribution drift, label drift, and business KPI correlation on an ongoing basis.

---

## ML Pipeline Architecture

```
Data Ingestion → Validation → Preprocessing → Feature Engineering →
Model Training → Evaluation → Artifact Registration → Deployment → Monitoring
         ^                                                              |
         |______________________ Retraining Loop ______________________|
```

---

## Project Structure

```
ml-project/
├── data/
│   ├── raw/                      # Immutable source data
│   ├── processed/                # Cleaned, split datasets
│   └── features/                 # Engineered feature sets
├── models/
│   ├── trained/                  # Serialized model artifacts
│   └── experiments/              # MLflow experiment runs
├── notebooks/
│   └── exploration.ipynb         # EDA only — no production logic
├── src/
│   ├── data/
│   │   ├── ingestion.py
│   │   ├── validation.py         # Schema + quality checks
│   │   └── preprocessing.py
│   ├── features/
│   │   └── feature_engineering.py
│   ├── models/
│   │   ├── train.py
│   │   ├── predict.py
│   │   └── evaluate.py
│   ├── serving/
│   │   ├── api.py                # FastAPI inference server
│   │   └── batch.py              # Batch prediction runner
│   └── monitoring/
│       └── drift_detector.py
├── config/
│   └── config.yaml
├── tests/
│   ├── test_features.py
│   ├── test_model.py
│   └── test_serving.py
├── mlflow.yaml                   # Experiment tracking config
├── requirements.txt
└── Dockerfile
```

---

## Data Validation Protocol

Always validate before training. Fail loudly, not silently.

```python
# src/data/validation.py
import pandas as pd
import numpy as np
from dataclasses import dataclass, field
from typing import Any

@dataclass
class ValidationResult:
    passed: bool
    issues: list[str] = field(default_factory=list)

def validate_dataset(df: pd.DataFrame, schema: dict[str, Any]) -> ValidationResult:
    issues = []

    # 1. Schema check
    for col, dtype in schema.items():
        if col not in df.columns:
            issues.append(f"Missing column: {col}")
        elif not pd.api.types.is_dtype_equal(df[col].dtype, dtype):
            issues.append(f"Column '{col}' expected {dtype}, got {df[col].dtype}")

    # 2. Null audit
    null_counts = df.isnull().sum()
    for col, count in null_counts[null_counts > 0].items():
        pct = count / len(df) * 100
        if pct > 5:
            issues.append(f"Column '{col}' has {pct:.1f}% nulls — investigate before imputation")

    # 3. Class imbalance check (classification)
    if "target" in df.columns:
        counts = df["target"].value_counts(normalize=True)
        minority_pct = counts.min() * 100
        if minority_pct < 5:
            issues.append(f"Severe class imbalance: minority class is {minority_pct:.1f}% — consider oversampling or class weights")

    # 4. Train/test leakage check (caller responsibility)
    # Ensure index is reset and no temporal leak exists before splitting

    return ValidationResult(passed=len(issues) == 0, issues=issues)
```

---

## Training Pipeline (Classical ML)

```python
# src/models/train.py
import pandas as pd
import numpy as np
import mlflow
import mlflow.sklearn
from sklearn.model_selection import train_test_split, cross_val_score
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import classification_report, roc_auc_score
import joblib

class ModelTrainer:
    def __init__(self, config: dict):
        self.config = config
        self.model = None

    def load_data(self):
        df = pd.read_csv(self.config["data_path"])
        X = df.drop("target", axis=1)
        y = df["target"]
        return train_test_split(
            X, y,
            test_size=self.config.get("test_size", 0.2),
            random_state=self.config.get("random_state", 42),
            stratify=y,  # maintain class distribution
        )

    def train(self, X_train, y_train):
        with mlflow.start_run():
            mlflow.log_params(self.config)

            self.model = RandomForestClassifier(
                n_estimators=self.config.get("n_estimators", 100),
                max_depth=self.config.get("max_depth", 10),
                class_weight="balanced",  # handles imbalanced classes
                random_state=self.config.get("random_state", 42),
                n_jobs=-1,
            )
            self.model.fit(X_train, y_train)

            # Cross-validate on training set for stable metric estimates
            cv_scores = cross_val_score(self.model, X_train, y_train, cv=5, scoring="f1_weighted")
            mlflow.log_metric("cv_f1_mean", cv_scores.mean())
            mlflow.log_metric("cv_f1_std", cv_scores.std())

            mlflow.sklearn.log_model(self.model, "model")
            return self.model

    def evaluate(self, X_test, y_test) -> dict:
        predictions = self.model.predict(X_test)
        probas = self.model.predict_proba(X_test)
        report = classification_report(y_test, predictions, output_dict=True)
        auc = roc_auc_score(y_test, probas[:, 1])

        mlflow.log_metric("test_accuracy", report["accuracy"])
        mlflow.log_metric("test_f1_weighted", report["weighted avg"]["f1-score"])
        mlflow.log_metric("test_auc_roc", auc)
        return {**report, "auc_roc": auc}

    def save_model(self, path: str):
        joblib.dump(self.model, path)
```

---

## LLM/GenAI Pipeline Patterns

### Pattern 1: LLM Classification with Structured Output

```python
# src/models/llm_classifier.py
import anthropic
import json
from pydantic import BaseModel

client = anthropic.Anthropic()

class ClassificationResult(BaseModel):
    label: str
    confidence: str  # "high" | "medium" | "low"
    reasoning: str

SYSTEM_PROMPT = """You are a document classifier. Classify the input text into exactly one category.
Categories: LEGAL, FINANCIAL, TECHNICAL, GENERAL

Respond with valid JSON only:
{"label": "<category>", "confidence": "<high|medium|low>", "reasoning": "<one sentence>"}"""

def classify_with_llm(text: str, model: str = "claude-haiku-3-5-20251001") -> ClassificationResult:
    """Use LLM for zero-shot or few-shot classification.
    Prefer Haiku for throughput-sensitive classification tasks.
    """
    response = client.messages.create(
        model=model,
        max_tokens=256,
        system=SYSTEM_PROMPT,
        messages=[{"role": "user", "content": text[:4000]}],  # truncate to avoid OOM
    )
    raw = response.content[0].text
    data = json.loads(raw)
    return ClassificationResult(**data)
```

### Pattern 2: LLM Evaluation Harness (Karpathy Loop)

Build an evaluation loop that measures quality quantitatively before shipping:

```python
# src/evaluation/llm_eval.py
import json
from pathlib import Path
from dataclasses import dataclass, field

@dataclass
class EvalResult:
    total: int = 0
    correct: int = 0
    failures: list[dict] = field(default_factory=list)

    @property
    def accuracy(self) -> float:
        return self.correct / self.total if self.total > 0 else 0.0

def run_eval_harness(eval_dataset_path: str, predict_fn, expected_key: str = "label") -> EvalResult:
    """
    Karpathy eval pattern: run every example, record failures, compute aggregate metric.
    Never ship a prompt change without running this first.
    """
    dataset = json.loads(Path(eval_dataset_path).read_text())
    result = EvalResult(total=len(dataset))

    for example in dataset:
        prediction = predict_fn(example["input"])
        expected = example[expected_key]
        if prediction == expected:
            result.correct += 1
        else:
            result.failures.append({
                "input": example["input"],
                "expected": expected,
                "got": prediction,
            })

    return result

# Usage
result = run_eval_harness("data/eval/classification_eval.json", classify_with_llm)
print(f"Accuracy: {result.accuracy:.2%} ({result.correct}/{result.total})")
if result.accuracy < 0.90:
    raise ValueError(f"Accuracy {result.accuracy:.2%} below threshold 90% — do not deploy")
```

---

## Hyperparameter Tuning

```python
# src/models/hparam_search.py
import optuna
import mlflow

def objective(trial: optuna.Trial, X_train, y_train) -> float:
    """Optuna objective — returns the metric to maximize."""
    params = {
        "n_estimators": trial.suggest_int("n_estimators", 50, 500),
        "max_depth": trial.suggest_int("max_depth", 3, 20),
        "min_samples_leaf": trial.suggest_int("min_samples_leaf", 1, 20),
        "max_features": trial.suggest_categorical("max_features", ["sqrt", "log2"]),
    }
    with mlflow.start_run(nested=True):
        mlflow.log_params(params)
        model = RandomForestClassifier(**params, random_state=42, n_jobs=-1)
        scores = cross_val_score(model, X_train, y_train, cv=5, scoring="f1_weighted")
        mlflow.log_metric("cv_f1_mean", scores.mean())
    return scores.mean()

# Run: optuna.create_study(direction="maximize").optimize(lambda t: objective(t, X, y), n_trials=100)
```

---

## Model Serving

```python
# src/serving/api.py
from fastapi import FastAPI, HTTPException
from pydantic import BaseModel
import joblib
import numpy as np
import time

app = FastAPI()
model = joblib.load("models/trained/model.joblib")

class PredictionRequest(BaseModel):
    features: list[float]

class PredictionResponse(BaseModel):
    prediction: int
    probability: float
    latency_ms: float

@app.post("/predict", response_model=PredictionResponse)
def predict(request: PredictionRequest):
    if len(request.features) == 0:
        raise HTTPException(status_code=422, detail="features must not be empty")
    start = time.perf_counter()
    features = np.array(request.features).reshape(1, -1)
    prediction = int(model.predict(features)[0])
    probability = float(model.predict_proba(features)[0].max())
    latency_ms = (time.perf_counter() - start) * 1000
    return PredictionResponse(prediction=prediction, probability=probability, latency_ms=latency_ms)

@app.get("/health")
def health():
    return {"status": "ok"}
```

---

## Production Monitoring: Data Drift Detection

```python
# src/monitoring/drift_detector.py
import numpy as np
from scipy.stats import ks_2samp

def detect_feature_drift(
    reference: np.ndarray,
    current: np.ndarray,
    feature_name: str,
    threshold: float = 0.05,
) -> dict:
    """
    Kolmogorov-Smirnov test for distribution drift.
    p-value < threshold means the distributions are significantly different.
    """
    statistic, p_value = ks_2samp(reference, current)
    return {
        "feature": feature_name,
        "ks_statistic": round(statistic, 4),
        "p_value": round(p_value, 4),
        "drift_detected": p_value < threshold,
    }
```

---

## Feature Engineering

```python
# src/features/feature_engineering.py
from sklearn.base import BaseEstimator, TransformerMixin
import pandas as pd
import numpy as np

class FeatureEngineer(BaseEstimator, TransformerMixin):
    """Sklearn-compatible transformer for consistent train/serve feature computation."""

    def fit(self, X, y=None):
        # Compute statistics only from training data to avoid leakage
        if "amount" in X.columns:
            self._amount_mean = X["amount"].mean()
            self._amount_std = X["amount"].std()
        return self

    def transform(self, X):
        X = X.copy()

        if "date" in X.columns:
            dt = pd.to_datetime(X["date"])
            X["year"] = dt.dt.year
            X["month"] = dt.dt.month
            X["day_of_week"] = dt.dt.dayofweek
            X = X.drop(columns=["date"])

        if "amount" in X.columns:
            X["amount_log"] = np.log1p(X["amount"].clip(lower=0))
            # Use training-set statistics (no leakage)
            X["amount_zscore"] = (X["amount"] - self._amount_mean) / (self._amount_std + 1e-9)

        if "description" in X.columns:
            X["text_length"] = X["description"].str.len().fillna(0)
            X["word_count"] = X["description"].str.split().str.len().fillna(0)
            X = X.drop(columns=["description"])

        return X
```

---

## Blocking Violations (NEVER)

| Violation | Consequence | Recovery |
|---|---|---|
| Training on full dataset without held-out test set | Inflated metrics — cannot detect overfitting; deployed model has far lower real-world accuracy | Reserve stratified holdout before training; cross-validate |
| Deploying model without rollback path | Accuracy degradation requires hours-long redeploy | Register model versions with traffic-switching mechanism |
| Hardcoding prompts in production without versioning | Untracked prompt changes cause invisible behavior regressions | Version prompts in VCS with test suite |
| Using accuracy as sole metric for imbalanced data | 95% accuracy by predicting majority class; zero recall for minority cases | Add precision, recall, F1, and confusion matrix |
| Skipping baseline benchmark | Complex model provides no evidence of value over simple heuristic | Always benchmark baseline before training |
| Training on features computed with future data | Massive eval inflation; deployed model receives only past data and performs much worse | Validate no temporal leakage in feature pipeline |

## Verification

Before declaring a model or pipeline complete:

### Self-Verification Checklist

- [ ] A quantitative success metric is defined and logged (accuracy, F1, AUC, ROUGE, etc.)
- [ ] An eval harness exists and passes at the defined threshold (e.g. >90% accuracy)
- [ ] All experiments are tracked in MLflow (or equivalent) with parameters and metrics
- [ ] Training data and test data are strictly separated; no temporal or row-level leakage
- [ ] The model serves predictions via a health-checked endpoint or batch runner
- [ ] At least one drift detection or monitoring hook is in place for production
- [ ] The `requirements.txt` or `pyproject.toml` pins exact dependency versions
- [ ] A baseline (simple heuristic or existing model) has been benchmarked and recorded
- [ ] Data leakage check: training set and validation/test set have 0 overlapping sample IDs — `SELECT COUNT(*) FROM train JOIN test ON train.id = test.id` returns 0
- [ ] Model serialisation round-trip verified: saved model loaded and produces identical predictions on 10 held-out samples — diff of outputs = 0

### Verification Commands

```bash
# Run eval harness
python -m src.evaluation.llm_eval

# Check data leakage
python -c "from src.data.validation import check_leakage; check_leakage('train.csv', 'test.csv')"

# Verify model serialization
python -c "from src.models.train import verify_roundtrip; verify_roundtrip('models/trained/model.joblib')"

# Run drift detection
python -m src.monitoring.drift_detector --reference data/processed/reference.parquet --current data/processed/current.parquet
```

### Quality Gates

| Gate | Criteria | Fail Action |
|---|---|---|
| Success Metric | Meets or exceeds agreed threshold on held-out test set | Do not deploy. Re-train or adjust threshold. |
| Reproducibility | Re-running `train.py` with same config & data produces same metrics within tolerance | Debug non-determinism (race condition, random seed, data loading order) |
| Eval Harness | Runs in CI and fails loudly if model quality regresses | Block PR. Revert model change. |
| Monitoring | Alerts configured for prediction drift and KPI deviation | Add before deploying to production. |
| Latency Budget | P95 latency within budget on production-equivalent hardware | Optimize model or serving infrastructure. |

## Performance & Cost

### Model Selection

| Task Complexity | Recommended Tool | Estimated Cost |
|---|---|---|
| Simple classification (<10K samples) | sklearn / XGBoost | Minimal (CPU) |
| Deep learning (large dataset) | PyTorch / transformers | GPU hours |
| LLM classification / extraction | API call (Haiku) | $0.80/1M input tokens |
| LLM complex reasoning | API call (Sonnet/Opus) | $3-15/1M tokens |

### Parallelization

- **Hyperparameter search:** Run N trials in parallel if GPU/CPU budget allows
- **LLM batch classification:** Can parallelize independent calls; watch rate limits
- **Training + eval:** Sequential — eval depends on trained model

### Context Budget

- **Expected context usage:** 5-10KB per model development session
- **When to context-optimize:** When reviewing >5 experiment runs or training logs
- **Context recovery:** Use `ctx_execute` for training logs, `ctx_execute_file` for large datasets

## Examples

### Example 1: Classification Model with Baseline

**User request:** "Build a model to classify support tickets by urgency."

**Skill execution:**
1. Define success metric: F1-weighted on urgency classes
2. Benchmark baseline: zero-shot LLM classifier (Haiku) → 72% F1
3. Build sklearn pipeline: TF-IDF + LogisticRegression → 78% F1 (beats baseline ✓)
4. Track experiment in MLflow with parameters and metrics
5. Validate data: check class imbalance (urgent=3% — apply class_weight="balanced")
6. Deploy as FastAPI endpoint with health check
7. Add drift detection monitoring

**Result:** Production-classifier with known baseline comparison, experiment tracking, and drift monitoring.

### Example 2: LLM Evaluation Harness

**User request:** "We changed the prompt for our summarization system. How do we know it's better?"

**Skill execution:**
1. Create eval dataset with 50 input-summary pairs from gold-standard human labels
2. Run both old and new prompts through `run_eval_harness()` using ROUGE-L scoring
3. Old prompt: ROUGE-L=0.72. New prompt: ROUGE-L=0.68 (regression!)
4. Revert prompt change. Log failure cases for analysis.
5. Document: the new prompt introduced verbosity that reduced lexical overlap

**Result:** Regression caught before deployment. Eval harness prevents quality degradation.

### Example 3: Edge Case — Train/Test Leakage

**User request:** "My model gets 98% accuracy in eval but 60% in production."

**Skill execution:**
1. Check data splitting: is `train_test_split` called before any feature engineering?
2. Found: feature engineering computed global statistics before split, leaking test distribution
3. Fix: fit `FeatureEngineer` on training data only, transform test data separately
4. Re-run: accuracy drops to 76% (honest number) — still beats baseline (72%)

**Result:** Honest evaluation. Model performance matches production expectations.

## Anti-Patterns

- Never train on the full dataset without a held-out test set because cross-validation estimates in-sample performance and cannot detect overfitting to the training distribution — without a truly unseen holdout, you will report inflated metrics and deploy a model whose real-world accuracy is significantly lower than measured.
- Never deploy a model without a rollback path because a sudden accuracy degradation or prediction distribution shift in production requires the ability to restore the previous model version within minutes — without registered versions and a traffic-switching mechanism, the only recovery option is a full redeploy that takes hours.
- Never hardcode a prompt in production without versioning it because prompts are code that controls model behavior, and an untracked inline string literal cannot be rolled back, A/B tested, or audited after a behavior regression — the prompt that caused the incident becomes invisible.
- Never use accuracy as the sole metric for imbalanced datasets because a model that predicts the majority class for every sample achieves 95% accuracy on a 5% minority class problem while having zero recall for the cases that matter most, and publishing that accuracy metric gives stakeholders a false picture of model quality.
- Never call an LLM API in a tight loop without rate limiting and cost tracking because an unbounded loop over thousands of records can exhaust the monthly API budget in minutes and trigger rate limit errors that fail silently if not handled, corrupting partial results without any alert.
- Never skip the baseline because a complex model that does not significantly outperform a majority-class predictor or a simple heuristic provides no evidence of value and adds operational maintenance cost, model drift risk, and infrastructure overhead for no measurable gain.
- Never train on features computed using future data because leaking target-correlated information from after the prediction timestamp inflates evaluation metrics dramatically — when deployed, the model receives only past data and its real performance will be far below the training-time numbers, causing silent accuracy degradation that only surfaces weeks after launch.

---

## Failure Modes

| Situation                                   | Response                                                                                                                                    |
| ------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------- |
| GPU out of memory during training           | Reduce batch size, enable gradient checkpointing, switch to mixed precision (fp16). Profile memory with `torch.cuda.memory_summary()`.      |
| Training loss not converging                | Check learning rate (try 1e-4 to 1e-2 range), verify data normalization, inspect gradient norms for explosion/vanishing.                    |
| Model accuracy degraded after deployment    | Run drift detection on input features. Compare recent prediction distribution vs. training distribution. Check for schema changes upstream. |
| LLM outputs inconsistent JSON               | Add output validation + retry with explicit JSON-mode or structured output API parameter. Log raw outputs for debugging.                    |
| Evaluation metric inflated                  | Check for train/test leakage. Verify stratified split was used. Re-run with a fresh random seed.                                            |
| MLflow experiment not found                 | Ensure `MLFLOW_TRACKING_URI` is set. Check run IDs are not nested under an already-closed parent run.                                       |
| Class imbalance causes poor minority recall | Apply `class_weight="balanced"`, oversample with SMOTE, or adjust decision threshold post-training using precision-recall curve.            |
| Data pipeline silently drops rows           | Add row-count assertions at each stage. Log `df.shape` before and after each transform.                                                     |

---

## References

### Internal Dependencies
- `data-engineer` — Provides clean, validated data for model training
- `search-vector-architect` — Provides embedding and RAG infrastructure
- `cost-aware-llm-pipeline` — Routes model calls to cheapest viable model
- `observability-specialist` — Configures monitoring dashboards for model health
- `test-driven-development` — Writes test-first model evaluation harnesses

### External Standards
- [MLflow](https://mlflow.org/) — Experiment tracking and model registry
- [Optuna](https://optuna.org/) — Hyperparameter optimization framework
- [MTEB Leaderboard](https://huggingface.co/spaces/mteb/leaderboard) — Embedding model benchmark
- [Karpathy Eval Pattern](https://karpathy.github.io/2023/03/11/evals/) — Structured evaluation loop for AI systems

### Related Skills
- `data-engineer` — Precedes ml-engineer in the Data vertical chain
- `search-vector-architect` — Partner skill for embedding and retrieval work
- `cost-aware-llm-pipeline` — Companion skill for LLM cost optimization
- `observability-specialist` — Follows ml-engineer for production monitoring

## Changelog

| Version | Date | Changes |
|---|---|---|
| 2.0.0 | 2026-07-09 | Upgraded to Gold Standard v2.0: added frontmatter version/category/dependencies, Blocking Violations table, Verification with commands/quality gates, Performance & Cost section, Examples, References, Changelog. Reorganized to 12-section template. |

---

## Integration with Mega-Mind

`ml-engineer` sits in the Data domain of the skill routing matrix. It is invoked by `mega-mind` when the request involves model training, evaluation, or deployment. For complex ML features, it is preceded by `search-first` (check for existing libraries/APIs) and `data-engineer` (schema design, ETL). Output artifacts (trained model, eval results) feed into `deployment-patterns` for production rollout. For LLM-specific cost control, always coordinate with `cost-aware-llm-pipeline`.

**Chain:** `search-first` → `data-engineer` → `ml-engineer` → `deployment-patterns` → `observability-specialist`