Mlops Quickstart Overview

Databricks MLOps Quickstart — Overview

This repository is an end-to-end MLOps template on Databricks. It uses the Iris classification dataset as a placeholder — it is meant to be adapted to any dataset and ML problem type (regression, clustering, classification, forecasting, etc.).

When the user wants to adapt a specific area, load the matching skill:

Repository Structure

notebooks/
  1_data_preprocessing/     → Data ingestion & feature engineering
  2_model_training_and_deployment/
    model_training.ipynb    → Train & register model in Unity Catalog
    model_deployment/       → MLflow 3 deployment pipeline (evaluate → approve → deploy)
  3_inference/
    batch_inference.ipynb   → Batch scoring with the Champion model
    realtime_inference.ipynb → Serving endpoint example
resources/                  → Databricks Asset Bundle job definitions (one YAML per job)
databricks.yml              → Bundle config with dev/prod targets and shared variables
azure_pipelines.yml         → CD pipeline for Azure DevOps
.github/workflows/          → GitHub Actions CD pipeline (alternative)

Parameterization Contract

All notebooks accept parameters via dbutils.widgets. Three core parameters are wired through every notebook and job:

Rules to keep this contract intact:

• Three-level Unity Catalog references always use {catalog_name}.{schema_name}.<object> — never hardcode a catalog or schema inside notebook code.
• Job YAMLs in resources/ declare these as job-level parameters sourced from ${var.*} bundle variables, then pass them to notebooks via base_parameters.
• If a notebook needs a new parameter, add it in three places: the widget in the notebook, the job parameters block, and the task base_parameters.

Key Conventions

• Challenger/Champion pattern: newly trained models are registered with the @challenger alias. After passing evaluation and approval, the deployment notebook promotes them to @champion. Inference notebooks always load the @champion alias.
• MLflow experiment naming: experiments are scoped per user and catalog — /{user}/{model}_{catalog} — to avoid collisions during development.
• Serving endpoint naming: derived as {catalog_name}-{schema_name}-{model}-endpoint (dots replaced with dashes) to ensure valid endpoint names.
• Idempotent notebooks: ingestion and inference notebooks check for table existence before deciding whether to create vs. append/overwrite. Preserve this behavior when adapting.
• Default compute: jobs without an explicit job_clusters block run on serverless. Dedicated job compute can be enabled by uncommenting the job_clusters section in the job YAML.
• Pinned dependencies: every library version is pinned in requirements.txt at the repo root. Each notebook installs from it with a single %pip install -r ../../requirements.txt (or ../../../ for deployment notebooks). Never pin a version inline. See manage-dependencies.

Adaptation Workflow

When a user clones this repo to build their own ML solution, work through the adaptation areas in this order — and load the matching skill for each step:

1. Data → adapt-data-ingestion
2. Model training → adapt-model-training
3. Deployment pipeline → adapt-model-deployment
4. Inference → adapt-inference
5. Bundle config + CI/CD → adapt-bundle-and-cicd

Each adapt-* skill is self-contained for its area but assumes the conventions in this overview hold.