jcdavis131 Dependency Hygiene
A single heavy dependency can hold a whole monorepo's environment hostage via its transitive pins. The fix is structural, not behavioral: keep heavy adapters out of the universal lock.
The rule
If a dependency meets any of these, it belongs in an opt-in extra, not the core deps:
- Heavy install (large wheels, model downloads, system libs).
- Transitive pins that constrain foundational libs (tokenizers, transformers, torch, numpy, pillow).
- Only one workspace / one code path actually needs it.
- It pulls a pin that conflicts with the training or inference environment.
Common offenders: litellm, instructor, crawl4ai, marker-pdf, outlines, transformers, torch, sentence-transformers.
Pattern (PEP 621 / pyproject)
[project]
name = "my-package"
dependencies = ["pypdf>=4.0"] # always needed, light, no conflicting pins
[project.optional-dependencies]
llm = ["litellm>=1.40", "instructor>=1.3"] # heavy, pin-pulling → opt-in
pdf = ["pypdf>=4.0"]
vector = ["qdrant-client>=1.9"]
Install only where needed:
uv pip install -e ".[llm]" # the workspace that talks to LLMs
uv pip install -e "." # everyone else — stays coherent
Write the "why" into the diff
The next agent (or future-you) will see the extra and be tempted to promote it to core deps "for convenience". Stop that with a comment in the pyproject:
# Heavy adapters (litellm, instructor) are opt-in via `uv pip install` —
# kept out of the universal lock so their transitive pins
# (litellm->tokenizers) can't constrain the shared training environment.
A one-line rationale in the same file as the decision is the cheapest possible guardrail.
Anti-patterns
- "Just add it to core deps, it's easier." — one heavy lib in core deps and now every CI job, every worker, every training run inherits its pins.
- Extras without a comment. The next contributor re-inlines them.
- One giant
extras = "all"that re-combines everything — defeats the isolation. - Promoting an extra back to core without checking the transitive pin set. Run
uv treeorpip showfirst.