majiayu000 MLflow
MLflow is the standard for tracking experiments. v3.0 (2025) pivots to GenAI, adding LLM Tracing, Prompt Management, and "LLM-as-a-Judge".
When to Use
- Experiment Tracking: Logging hyperparameters (
lr=0.01) and metrics (accuracy=0.98). - GenAI Tracing: Visualizing the full chain of a RAG application.
- Model Registry: Versioning models (
my-model/v3) for deployment.
Core Concepts
Tracking URI
Where logs are stored (local ./mlruns or remote http://mlflow-server).
Autologging
mlflow.autolog() automatically captures params from Scikit-learn, PyTorch, etc.
LLM Tracing
OpenTelemetry-based tracing to debug prompt chains.
Best Practices (2025)
Do:
- Use
mlflow.evaluate(): To run "LLM-as-a-Judge" metrics on your RAG pipeline. - Use Prompt Engineering UI: MLflow 3.0 has a UI to iterate on prompts.
Don't:
- Don't use it for data storage: Log artifacts (models), not datasets. Log metadata about datasets instead.