MLflow Onboarding
MLflow supports two broad use cases that require different onboarding paths:
- GenAI applications and agents: LLM-powered apps, chatbots, RAG pipelines, tool-calling agents. Key MLflow features include tracing for observability, evaluation with LLM judges, and prompt management, among others.
- Traditional ML / deep learning models: scikit-learn, PyTorch, TensorFlow, XGBoost, etc. Key MLflow features include experiment tracking (parameters, metrics, artifacts), model logging, and model deployment, among others.
Determining which use case applies is the first and most important step. The onboarding path, quickstart tutorials, and integration steps differ significantly between the two.
Step 1: Determine the Use Case
Before recommending tutorials or integration steps, determine which use case the user is working on. Use the signals below, checking them in order. If the signals are ambiguous or absent, you MUST ask the user directly.
Signal 1: Check the Codebase
Search the user's project for imports and usage patterns that indicate the use case:
GenAI indicators (any of these suggest GenAI):
- Imports from LLM client libraries:
openai, anthropic, google.generativeai, google.genai, langchain, langchain_openai, langgraph, llamaindex, litellm, autogen, crewai, dspy
- Imports from MLflow GenAI modules:
mlflow.genai, mlflow.tracing, mlflow.openai, mlflow.langchain
- Usage of chat completions, embeddings, or agent frameworks
- Prompt templates or prompt engineering code
Traditional ML indicators (any of these suggest ML):
- Imports from ML frameworks:
sklearn, torch, tensorflow, keras, xgboost, lightgbm, catboost, statsmodels, scipy
- Imports from MLflow ML modules:
mlflow.sklearn, mlflow.pytorch, mlflow.tensorflow
- Model training loops,
.fit() calls, hyperparameter tuning code
- Dataset loading with tabular/image/time-series data
# Search for GenAI indicators
grep -rl --include='*.py' -E '(import openai|import anthropic|from langchain|from langgraph|import litellm|from mlflow\.genai|from mlflow\.tracing|mlflow\.openai|mlflow\.langchain|ChatCompletion|chat\.completions)' .
# Search for ML indicators
grep -rl --include='*.py' -E '(from sklearn|import torch|import tensorflow|import keras|import xgboost|import lightgbm|mlflow\.sklearn|mlflow\.pytorch|mlflow\.tensorflow|\.fit\()' .
Signal 2: Check the Experiment Type Tag
If the codebase or project directory is the MLflow repository itself, skip to Signal 3 — the MLflow repo contains code for all use cases and does not indicate the user's intent.
If the experiment ID is known, check its mlflow.experimentKind tag. This tag is set by MLflow to indicate the experiment type:
mlflow experiments get --experiment-id <EXPERIMENT_ID> --output json > /tmp/exp_detail.json
jq -r '.tags["mlflow.experimentKind"] // "not set"' /tmp/exp_detail.json
genai_development → GenAI use case
custom_model_development → Traditional ML use case
- Not set → Proceed to Signal 3
If the experiment ID is not known, skip to Signal 3.
Signal 3: Ask the User
If the codebase and experiment signals are inconclusive, ask directly:
Are you building a GenAI application (e.g., an LLM-powered chatbot, RAG pipeline, or tool-calling agent) or a traditional ML/deep learning model (e.g., training a classifier, regression model, or neural network)?
Do not guess. The onboarding paths are different enough that starting down the wrong one wastes the user's time.
Step 2: Recommend Quickstart Tutorials
Once the use case is determined, recommend the appropriate quickstart tutorials from the MLflow documentation. Present them to the user and ask if they'd like to follow along or jump directly to integrating MLflow into their project.
GenAI Path
The MLflow GenAI documentation is at: https://mlflow.org/docs/latest/genai/getting-started/
Choose the most relevant tutorials based on the user's context and what they've told you. Available tutorials include:
If none of these match the user's needs, look up the MLflow GenAI documentation for more relevant guides.
Traditional ML Path
The MLflow ML documentation is at: https://mlflow.org/docs/latest/ml/getting-started/
Choose the most relevant tutorials based on the user's context and what they've told you. Available tutorials include:
If none of these match the user's needs, look up the MLflow ML documentation for more relevant guides.
Step 3: Integrate MLflow into the User's Project
After the user has reviewed the quickstart tutorials (or opted to skip them), offer to help integrate MLflow directly into their codebase. Always ask for the user's consent before making changes to their code.
GenAI Integration
The core integration for GenAI apps is tracing — capturing LLM calls, tool invocations, and agent steps automatically.
If asked to create an example project: Do not assume the user has LLM API keys (e.g., OpenAI, Anthropic). Instead, create traces with mock data using @mlflow.trace and mlflow.start_span() to demonstrate tracing without requiring external API access. For example:
import mlflow
mlflow.set_experiment("example-genai-app")
@mlflow.trace
def mock_chat(query: str) -> str:
with mlflow.start_span(name="retrieve_context") as span:
context = "Mock retrieved context for: " + query
span.set_inputs({"query": query})
span.set_outputs({"context": context})
with mlflow.start_span(name="generate_response") as span:
response = "Mock response based on: " + context
span.set_inputs({"context": context, "query": query})
span.set_outputs({"response": response})
return response
mock_chat("What is MLflow?")
What to set up (for an existing project):
Autologging — If the user's code uses a supported framework, a single line automatically traces all calls to their LLM provider. See https://mlflow.org/docs/latest/genai/tracing/ for the full list of supported providers. If the provider is supported:
import mlflow
# Pick the one that matches the user's LLM provider:
mlflow.openai.autolog() # OpenAI SDK
mlflow.anthropic.autolog() # Anthropic SDK
mlflow.gemini.autolog() # Google Gemini (google-genai SDK)
mlflow.langchain.autolog() # LangChain / LangGraph
mlflow.litellm.autolog() # LiteLLM
Add this call once at application startup (e.g., top of main.py, app.py, or the entry point module). It must execute before any LLM calls are made.
If the provider is not supported by autologging, skip to step 3 (Custom tracing) and use @mlflow.trace to manually instrument the relevant functions.
Experiment configuration — Set the experiment so traces are organized:
mlflow.set_experiment("my-genai-app")
Or via environment variable: export MLFLOW_EXPERIMENT_NAME="my-genai-app"
Custom tracing (optional) — For functions that aren't automatically traced (custom tools, business logic), use the @mlflow.trace decorator:
@mlflow.trace
def my_custom_tool(query: str) -> str:
# ... tool logic ...
return result
Where to add it: Find the application's entry point or initialization module and add the autologging call there. Search for the main LLM client instantiation (e.g., openai.OpenAI(), ChatOpenAI()) to find the right location.
Prompt Registry (optional) — MLflow can version, store, and load prompts so application code loads a prompt by URI instead of hard-coding the template.
import mlflow
# Register a prompt version
prompt_version = mlflow.genai.register_prompt(
name="my_prompt",
template="Answer the user's question: {{question}}",
)
print(prompt_version.uri) # prompts:/my_prompt/1
# Load it back in application code
prompt = mlflow.genai.load_prompt("prompts:/my_prompt/1")
On Databricks (Unity Catalog): prompts are stored under a UC catalog.schema, so name is a three-part catalog.schema.my_prompt and the workspace must have the Prompt Registry preview enabled (account admin -> Previews -> "Prompt Registry"). On success register_prompt logs a workspace UI link and auto-links the active experiment's Prompts tab to catalog.schema. If that tab is empty after registering, the active experiment was not auto-linked (older mlflow versions do not set this tag automatically). Set it manually:
mlflow.set_experiment_tag(
"mlflow.promptRegistryLocation",
"catalog.schema", # two-part: catalog.schema (no prompt name)
)
Traditional ML Integration
The core integration for ML is experiment tracking — capturing parameters, metrics, and models from training runs.
What to set up:
Autologging — If the user's code uses a supported framework, a single line automatically logs parameters, metrics, and models during training. See https://mlflow.org/docs/latest/ml/ for the full list of supported frameworks. If the framework is supported:
import mlflow
# Pick the one that matches the user's ML framework:
mlflow.sklearn.autolog() # scikit-learn
mlflow.pytorch.autolog() # PyTorch / PyTorch Lightning
mlflow.tensorflow.autolog() # TensorFlow / Keras
mlflow.xgboost.autolog() # XGBoost
mlflow.lightgbm.autolog() # LightGBM
Add this call once before training starts. It automatically captures model.fit() calls, logged metrics, and model artifacts.
If the framework is not supported by autologging, skip to step 3 (Manual logging) and use mlflow.log_param(), mlflow.log_metric(), and mlflow.log_artifact() to log data explicitly.
Experiment configuration — Set the experiment so runs are organized:
mlflow.set_experiment("my-ml-experiment")
Or via environment variable: export MLFLOW_EXPERIMENT_NAME="my-ml-experiment"
Manual logging (optional) — For metrics or parameters not captured by autologging:
with mlflow.start_run():
mlflow.log_param("custom_param", value)
mlflow.log_metric("custom_metric", value)
Where to add it: Find the training script or module where model.fit() (or equivalent) is called. Add the autologging call before the training loop begins.
Verification
After integration, verify that MLflow is capturing data correctly:
GenAI Verification
- Run the application and trigger at least one LLM call
- Check for traces:
mlflow traces search \
--experiment-id <EXPERIMENT_ID> \
--max-results 5 \
--extract-fields 'info.trace_id,info.state,info.request_time' \
--output json > /tmp/verify_traces.json
jq '.traces | length' /tmp/verify_traces.json
- If traces appear, open the MLflow UI to inspect them visually
ML Verification
- Run the training script
- Check for runs:
mlflow runs search \
--experiment-id <EXPERIMENT_ID> \
--max-results 5 \
--output json > /tmp/verify_runs.json
jq '.runs | length' /tmp/verify_runs.json
- If runs appear, open the MLflow UI to inspect logged parameters, metrics, and artifacts
1---
2name: mlflow-onboarding
3description: Onboards users to MLflow by determining their use case (GenAI agents/apps or traditional ML/deep learning) and guiding them through relevant quickstart tutorials and initial integration. If an experiment ID is available, it should be supplied as input to help determine the use case. Use when the user asks to get started with MLflow, set up tracking, add observability, or integrate MLflow into their project. Triggers on "get started with MLflow", "set up MLflow", "onboard to MLflow", "add MLflow to my project", "how do I use MLflow".
4metadata:
5 source: https://skillmd.com/skills/mlflow/mlflow-onboarding
6---
7
8# MLflow Onboarding
9
10MLflow supports two broad use cases that require different onboarding paths:
11
12- **GenAI applications and agents**: LLM-powered apps, chatbots, RAG pipelines, tool-calling agents. Key MLflow features include **tracing** for observability, **evaluation** with LLM judges, and **prompt management**, among others.
13- **Traditional ML / deep learning models**: scikit-learn, PyTorch, TensorFlow, XGBoost, etc. Key MLflow features include **experiment tracking** (parameters, metrics, artifacts), **model logging**, and **model deployment**, among others.
14
15Determining which use case applies is the first and most important step. The onboarding path, quickstart tutorials, and integration steps differ significantly between the two.
16
17## Step 1: Determine the Use Case
18
19Before recommending tutorials or integration steps, determine which use case the user is working on. Use the signals below, checking them in order. **If the signals are ambiguous or absent, you MUST ask the user directly.**
20
21### Signal 1: Check the Codebase
22
23Search the user's project for imports and usage patterns that indicate the use case:
24
25**GenAI indicators** (any of these suggest GenAI):
26- Imports from LLM client libraries: `openai`, `anthropic`, `google.generativeai`, `google.genai`, `langchain`, `langchain_openai`, `langgraph`, `llamaindex`, `litellm`, `autogen`, `crewai`, `dspy`
27- Imports from MLflow GenAI modules: `mlflow.genai`, `mlflow.tracing`, `mlflow.openai`, `mlflow.langchain`
28- Usage of chat completions, embeddings, or agent frameworks
29- Prompt templates or prompt engineering code
30
31**Traditional ML indicators** (any of these suggest ML):
32- Imports from ML frameworks: `sklearn`, `torch`, `tensorflow`, `keras`, `xgboost`, `lightgbm`, `catboost`, `statsmodels`, `scipy`
33- Imports from MLflow ML modules: `mlflow.sklearn`, `mlflow.pytorch`, `mlflow.tensorflow`
34- Model training loops, `.fit()` calls, hyperparameter tuning code
35- Dataset loading with tabular/image/time-series data
36
37```bash
38# Search for GenAI indicators
39grep -rl --include='*.py' -E '(import openai|import anthropic|from langchain|from langgraph|import litellm|from mlflow\.genai|from mlflow\.tracing|mlflow\.openai|mlflow\.langchain|ChatCompletion|chat\.completions)' .
40
41# Search for ML indicators
42grep -rl --include='*.py' -E '(from sklearn|import torch|import tensorflow|import keras|import xgboost|import lightgbm|mlflow\.sklearn|mlflow\.pytorch|mlflow\.tensorflow|\.fit\()' .
43```
44
45### Signal 2: Check the Experiment Type Tag
46
47If the codebase or project directory is the MLflow repository itself, skip to Signal 3 — the MLflow repo contains code for all use cases and does not indicate the user's intent.
48
49If the experiment ID is known, check its `mlflow.experimentKind` tag. This tag is set by MLflow to indicate the experiment type:
50
51```bash
52mlflow experiments get --experiment-id <EXPERIMENT_ID> --output json > /tmp/exp_detail.json
53jq -r '.tags["mlflow.experimentKind"] // "not set"' /tmp/exp_detail.json
54```
55
56- **`genai_development`** → GenAI use case
57- **`custom_model_development`** → Traditional ML use case
58- **Not set** → Proceed to Signal 3
59
60If the experiment ID is not known, skip to Signal 3.
61
62### Signal 3: Ask the User
63
64If the codebase and experiment signals are inconclusive, ask directly:
65
66> Are you building a **GenAI application** (e.g., an LLM-powered chatbot, RAG pipeline, or tool-calling agent) or a **traditional ML/deep learning model** (e.g., training a classifier, regression model, or neural network)?
67
68**Do not guess.** The onboarding paths are different enough that starting down the wrong one wastes the user's time.
69
70## Step 2: Recommend Quickstart Tutorials
71
72Once the use case is determined, recommend the appropriate quickstart tutorials from the MLflow documentation. Present them to the user and ask if they'd like to follow along or jump directly to integrating MLflow into their project.
73
74### GenAI Path
75
76The MLflow GenAI documentation is at: https://mlflow.org/docs/latest/genai/getting-started/
77
78Choose the most relevant tutorials based on the user's context and what they've told you. Available tutorials include:
79
80- **Tracing Quickstart** (https://mlflow.org/docs/latest/genai/tracing/quickstart/) — Enabling automatic tracing for LLM calls. Covers starting an MLflow server, creating an experiment, enabling autologging, and viewing traces in the UI.
81 - Python + OpenAI variant: https://mlflow.org/docs/latest/genai/tracing/quickstart/python-openai/
82 - TypeScript + OpenAI variant: https://mlflow.org/docs/latest/genai/tracing/quickstart/typescript-openai
83 - OpenTelemetry (language-agnostic) variant: also linked from the quickstart page
84- **Evaluation Quickstart** (https://mlflow.org/docs/latest/genai/eval-monitor/quickstart/) — Evaluating GenAI application quality using LLM judges (scorers). Covers defining datasets, prediction functions, and built-in + custom scorers.
85- **Version Tracking Quickstart** (https://mlflow.org/docs/latest/genai/version-tracking/quickstart/) — Prompt management, application versioning, and connecting tracing to versioned prompts.
86
87If none of these match the user's needs, look up the MLflow GenAI documentation for more relevant guides.
88
89### Traditional ML Path
90
91The MLflow ML documentation is at: https://mlflow.org/docs/latest/ml/getting-started/
92
93Choose the most relevant tutorials based on the user's context and what they've told you. Available tutorials include:
94
95- **Tracking Quickstart** (https://mlflow.org/docs/latest/ml/tracking/quickstart/) — Experiment tracking with scikit-learn: autologging, manual parameter/metric/model logging, and exploring results in the MLflow UI.
96- **Deep Learning Tutorial** (https://mlflow.org/docs/latest/ml/getting-started/deep-learning/) — Training a PyTorch model with MLflow logging: parameters, metrics, checkpoints, and system metrics (GPU utilization, memory).
97- **Hyperparameter Tuning Tutorial** (https://mlflow.org/docs/latest/ml/getting-started/hyperparameter-tuning/) — Running hyperparameter searches with Optuna + MLflow, comparing results, and selecting the best model.
98
99If none of these match the user's needs, look up the MLflow ML documentation for more relevant guides.
100
101## Step 3: Integrate MLflow into the User's Project
102
103After the user has reviewed the quickstart tutorials (or opted to skip them), offer to help integrate MLflow directly into their codebase. **Always ask for the user's consent before making changes to their code.**
104
105### GenAI Integration
106
107The core integration for GenAI apps is **tracing** — capturing LLM calls, tool invocations, and agent steps automatically.
108
109**If asked to create an example project:** Do not assume the user has LLM API keys (e.g., OpenAI, Anthropic). Instead, create traces with mock data using `@mlflow.trace` and `mlflow.start_span()` to demonstrate tracing without requiring external API access. For example:
110
111```python
112import mlflow
113
114mlflow.set_experiment("example-genai-app")
115
116@mlflow.trace
117def mock_chat(query: str) -> str:
118 with mlflow.start_span(name="retrieve_context") as span:
119 context = "Mock retrieved context for: " + query
120 span.set_inputs({"query": query})
121 span.set_outputs({"context": context})
122 with mlflow.start_span(name="generate_response") as span:
123 response = "Mock response based on: " + context
124 span.set_inputs({"context": context, "query": query})
125 span.set_outputs({"response": response})
126 return response
127
128mock_chat("What is MLflow?")
129```
130
131**What to set up (for an existing project):**
132
1331. **Autologging** — If the user's code uses a supported framework, a single line automatically traces all calls to their LLM provider. See https://mlflow.org/docs/latest/genai/tracing/ for the full list of supported providers. If the provider is supported:
134
135 ```python
136 import mlflow
137
138 # Pick the one that matches the user's LLM provider:
139 mlflow.openai.autolog() # OpenAI SDK
140 mlflow.anthropic.autolog() # Anthropic SDK
141 mlflow.gemini.autolog() # Google Gemini (google-genai SDK)
142 mlflow.langchain.autolog() # LangChain / LangGraph
143 mlflow.litellm.autolog() # LiteLLM
144 ```
145
146 Add this call once at application startup (e.g., top of `main.py`, `app.py`, or the entry point module). It must execute before any LLM calls are made.
147
148 If the provider is **not** supported by autologging, skip to step 3 (Custom tracing) and use `@mlflow.trace` to manually instrument the relevant functions.
149
1502. **Experiment configuration** — Set the experiment so traces are organized:
151
152 ```python
153 mlflow.set_experiment("my-genai-app")
154 ```
155
156 Or via environment variable: `export MLFLOW_EXPERIMENT_NAME="my-genai-app"`
157
1583. **Custom tracing** (optional) — For functions that aren't automatically traced (custom tools, business logic), use the `@mlflow.trace` decorator:
159
160 ```python
161 @mlflow.trace
162 def my_custom_tool(query: str) -> str:
163 # ... tool logic ...
164 return result
165 ```
166
167**Where to add it:** Find the application's entry point or initialization module and add the autologging call there. Search for the main LLM client instantiation (e.g., `openai.OpenAI()`, `ChatOpenAI()`) to find the right location.
168
1694. **Prompt Registry** (optional) — MLflow can version, store, and load prompts so application code loads a prompt by URI instead of hard-coding the template.
170
171 ```python
172 import mlflow
173
174 # Register a prompt version
175 prompt_version = mlflow.genai.register_prompt(
176 name="my_prompt",
177 template="Answer the user's question: {{question}}",
178 )
179 print(prompt_version.uri) # prompts:/my_prompt/1
180
181 # Load it back in application code
182 prompt = mlflow.genai.load_prompt("prompts:/my_prompt/1")
183 ```
184
185 **On Databricks (Unity Catalog):** prompts are stored under a UC `catalog.schema`, so `name` is a three-part `catalog.schema.my_prompt` and the workspace must have the Prompt Registry preview enabled (account admin -> Previews -> "Prompt Registry"). On success `register_prompt` logs a workspace UI link and auto-links the active experiment's Prompts tab to `catalog.schema`. If that tab is empty after registering, the active experiment was not auto-linked (older mlflow versions do not set this tag automatically). Set it manually:
186
187 ```python
188 mlflow.set_experiment_tag(
189 "mlflow.promptRegistryLocation",
190 "catalog.schema", # two-part: catalog.schema (no prompt name)
191 )
192 ```
193
194### Traditional ML Integration
195
196The core integration for ML is **experiment tracking** — capturing parameters, metrics, and models from training runs.
197
198**What to set up:**
199
2001. **Autologging** — If the user's code uses a supported framework, a single line automatically logs parameters, metrics, and models during training. See https://mlflow.org/docs/latest/ml/ for the full list of supported frameworks. If the framework is supported:
201
202 ```python
203 import mlflow
204
205 # Pick the one that matches the user's ML framework:
206 mlflow.sklearn.autolog() # scikit-learn
207 mlflow.pytorch.autolog() # PyTorch / PyTorch Lightning
208 mlflow.tensorflow.autolog() # TensorFlow / Keras
209 mlflow.xgboost.autolog() # XGBoost
210 mlflow.lightgbm.autolog() # LightGBM
211 ```
212
213 Add this call once before training starts. It automatically captures `model.fit()` calls, logged metrics, and model artifacts.
214
215 If the framework is **not** supported by autologging, skip to step 3 (Manual logging) and use `mlflow.log_param()`, `mlflow.log_metric()`, and `mlflow.log_artifact()` to log data explicitly.
216
2172. **Experiment configuration** — Set the experiment so runs are organized:
218
219 ```python
220 mlflow.set_experiment("my-ml-experiment")
221 ```
222
223 Or via environment variable: `export MLFLOW_EXPERIMENT_NAME="my-ml-experiment"`
224
2253. **Manual logging** (optional) — For metrics or parameters not captured by autologging:
226
227 ```python
228 with mlflow.start_run():
229 mlflow.log_param("custom_param", value)
230 mlflow.log_metric("custom_metric", value)
231 ```
232
233**Where to add it:** Find the training script or module where `model.fit()` (or equivalent) is called. Add the autologging call before the training loop begins.
234
235## Verification
236
237After integration, verify that MLflow is capturing data correctly:
238
239### GenAI Verification
240
2411. Run the application and trigger at least one LLM call
2422. Check for traces:
243 ```bash
244 mlflow traces search \
245 --experiment-id <EXPERIMENT_ID> \
246 --max-results 5 \
247 --extract-fields 'info.trace_id,info.state,info.request_time' \
248 --output json > /tmp/verify_traces.json
249 jq '.traces | length' /tmp/verify_traces.json
250 ```
2513. If traces appear, open the MLflow UI to inspect them visually
252
253### ML Verification
254
2551. Run the training script
2562. Check for runs:
257 ```bash
258 mlflow runs search \
259 --experiment-id <EXPERIMENT_ID> \
260 --max-results 5 \
261 --output json > /tmp/verify_runs.json
262 jq '.runs | length' /tmp/verify_runs.json
263 ```
2643. If runs appear, open the MLflow UI to inspect logged parameters, metrics, and artifacts