Manage Dependencies

Manage Dependencies

Target file: requirements.txt (repo root).

Convention

• requirements.txt is the single source of truth for every pinned Python library version used by the notebooks.

• Every notebook installs from it with one cell at the top:

  %pip install -r ../../requirements.txt        # 2 levels deep
  # or
  %pip install -r ../../../requirements.txt     # 3 levels deep (model_deployment/)
  dbutils.library.restartPython()
  

• Never pin a version inside a notebook. Always edit requirements.txt.

• Only pin libraries DBR does not already ship at a suitable version. Pinning a library DBR provides (scikit-learn, pandas, requests, numpy, …) at a different version than what DBR ships risks deserialization errors when the eval/inference notebooks load older MLflow model artifacts that were saved against the DBR-shipped version. The current pins are limited to mlflow (needed for the MLflow 3 deployment job pattern) and databricks-sdk (used by the serving endpoint code).

When to use

Use this skill whenever the user wants to:

• Add a new Python library the notebooks need.
• Upgrade or downgrade an existing pin (e.g. bump MLflow).
• Resolve a dependency conflict between requirements.txt and the Databricks Runtime / serverless environment.
• Remove a library the project no longer uses.

Step-by-step

1. Edit requirements.txt only. Keep entries pinned (pkg==X.Y.Z). Pinning ensures reproducible runs across dev, staging, and prod.

2. Match Databricks Runtime versions when possible. When the chosen pin conflicts with what the serverless environment or DBR ML provides, either:

   • Update the pin to match what the runtime ships, or
   • Confirm the pip resolver can install the requested version on top (some libraries can be upgraded, others are baked in).

3. Do not touch the %pip install -r … cells in notebooks. They use relative paths (../../requirements.txt from notebooks at depth 2, ../../../requirements.txt from notebooks at depth 3). If a new notebook is added, mirror the same pattern at the correct depth.

4. Verify locally where possible. Before deploying, run the affected notebook once interactively (or databricks bundle run <job>) so the pip install actually executes against the target compute.

Path depth reference

Edge cases

• Library bundled with DBR/serverless: pinning the same version it already provides is a no-op. Pinning a different version may fail if the library is "frozen" by the runtime. Test before merging.
• Transitive conflict: when pkg-a==X and pkg-b==Y disagree on a shared transitive dep, prefer pinning the transitive dep directly in requirements.txt rather than loosening the top-level pins.
• GPU / accelerated wheels: install from the correct index URL by adding --extra-index-url <url> at the top of requirements.txt (one per line, before any package lines).
• Private packages: workspace files or UC volumes can host private wheels — reference them with the full /Volumes/... or /Workspace/... path inside requirements.txt.
• Adding a brand-new notebook: when adding a notebook at a new depth, always include the install cell at the top with the correct relative path to requirements.txt.