Mlflow

MLflow Experiment Tracking AI Skill Guide

Overview & Engine Architecture

MLflow records experiments as runs with parameters, metrics, tags, and artifacts. A tracking server (or local mlruns/) stores metadata; the Model Registry promotes versions through stages/aliases. Agents log enough to reproduce a run (code version, data URI, params), avoid silent metric overwrites, and register only evaluated candidates.

Training code
   -> mlflow.start_run
   -> log_param / log_metric / log_artifact
   -> log_model -> Registry (aliases/stages)

When to use this skill

• Comparing hyperparameters and model families
• Attaching plots, confusion matrices, and conda/pip envs to runs
• Promoting models for batch/online inference with audit trail

Operational directives

1. Set experiment_name explicitly; do not dump everything into Default.
2. Log data version / query / path as params or tags.
3. Use nested runs for sweeps when parent/child clarity helps.
4. Prefer log_model with signature/input example for serving readiness.
5. Gate registry promotion on held-out metrics, not training loss alone.

Tracking example

import mlflow
import mlflow.sklearn
from mlflow.models import infer_signature

mlflow.set_tracking_uri("http://127.0.0.1:5000")
mlflow.set_experiment("churn-models")

with mlflow.start_run(run_name="hgb-v3") as run:
    mlflow.log_params({"max_depth": 6, "learning_rate": 0.08})
    # ... train sklearn pipeline `clf` ...
    mlflow.log_metrics({"roc_auc": 0.91, "f1": 0.74})
    sig = infer_signature(X_train, clf.predict(X_train))
    mlflow.sklearn.log_model(clf, artifact_path="model", signature=sig)
    mlflow.set_tags({"data": "s3://bucket/churn_v4.parquet", "git_sha": "abc123"})

CLI

mlflow ui --backend-store-uri sqlite:///mlflow.db --port 5000
mlflow runs list --experiment-name churn-models

Common failures

Best practices

• One primary metric per experiment for sorting.
• Log environment (mlflow.sklearn.autolog carefully - still review).
• Immutable training data snapshots or lake table versions.
• Pair with @scikit-learn / @pytorch / @huggingface-transformers.

Limitations

• Auth, multi-tenant servers, and artifact stores need ops setup.
• Autolog can miss custom loops - log manually when needed.
• Model serving deployment is separate from tracking.

Related skills

• @scikit-learn / @pytorch - training frameworks
• @airflow / @prefect - schedule training jobs that log to MLflow
• @jupyter - exploratory runs still should log when kept