Splitting Oversized Modules

Splitting oversized modules

A module nobody wants to open taxes every change in it, and agents pay that tax on every task because they do not carry knowledge between them. A 3000-line logic.py beside a 3000-line test_logic.py is roughly 55k input tokens of reading before a line gets written. Splitting one measured case cut the read-set 81 to 97% depending on the concern touched, while total lines grew about 5% from the repeated import headers.

So the goal is a small read-set for a typical change, not tidiness.

Is it worth doing here?

All three must hold: over roughly a thousand lines, several concerns that change independently, and something still actively changing in it. A big cohesive frozen file buys nothing. Skip generated files, and skip a file several people are mid-change in (git log --oneline -20 -- <file>).

A complexity warning on one function is a symptom of this, not a separate job. Extracting that function into helpers leaves every helper in the same file, so the read-set is unchanged and the file gets longer. Measure the file first (wc -l), and when it clears the gate, propose the split instead of the in-place extraction.

Splitting is a separate PR from whatever you came to do. Land the move as its own base PR and branch your work on top — see /stacking-prs. Never bundle it into a feature diff, and say what you are doing in one line before you start: which file, and that the split lands separately. Nobody minds the base PR; they mind finding it inside a feature diff.

If a human has not asked for the split, propose it and let them decide.

Method

Commands assume the repo root. S=.agents/skills/splitting-oversized-modules/scripts.

1. Map the symbols and assign them to concerns

uv run --no-project python $S/split_module.py <module> --skeleton > layout.json

Edit layout.json into modules named after concerns (baselines, quarantine, ci_status), not layers (helpers, utils, core). Put module-level state every module needs its own copy of — a logger, a compiled regex — under "__shared__".

Two rules while assigning:

Keep the dependency graph acyclic. If two modules need each other, the seam is wrong. Separating reads from writes fixes most cycles: a lifecycle module and a verification module that both need the same lookups should share a third leaf module holding those lookups.

Do not name a module after a common local variable. A module called artifacts or runs gets shadowed by artifacts = [...] inside a function, and Python binds the whole scope, so a call above the assignment fails too. ruff catches the reachable cases (F811, F823) but not one where the local is assigned before any module use. Pick a non-colliding name, or split finer until the name is specific.

2. Move the code

uv run --no-project python $S/split_module.py <module> layout.json
ruff check <package>/ --fix && ruff format <package>/

Hand-editing a 3000-line move loses code and silently rewrites it. The script copies each symbol verbatim by AST line range, refuses to run unless the layout covers every symbol exactly once, and requalifies cross-module references with tokenize so it never rewrites a name inside a docstring or string literal.

Cross-module calls come out as from . import baselines plus baselines.foo(...), never from .baselines import foo. One binding per symbol means a mock patch on the definition reaches every caller, and from . import x also survives an accidental cycle by falling back to sys.modules.

ruff --fix leaves an emptied if TYPE_CHECKING: pass behind. Delete those, then re-run ruff check --fix so the orphaned TYPE_CHECKING import goes too. The # --- Section --- dividers from the monolith are usually redundant once the module name says it: rg -n '^#\s*-{2,}' <package>/.

3. Retarget callers and mock patch targets

Every logic.foo() becomes <module>.foo(), and every patch target moves to the module that defines the symbol, since that is the binding its callers resolve:

patch("products.foo.backend.logic._post_commit_status")   # -> ...logic.ci_status._post_commit_status
patch.object(logic, "_post_commit_status")                # -> patch.object(ci_status, "_post_commit_status")

Sweep conftest.py too, not just test_*.py — fixtures patch, and a pass matched on test files leaves conftest pointing at paths that no longer exist. Enumerate candidates with rg -n 'patch\(|patch\.object\(' <tests>/. Watch for forms a naive sweep misses: a combined from x import a, logic, a symbol import (from ..logic import SomeError), and a name the old module merely re-exported, which should now come from its real source.

4. Split the tests the same way

A split module beside an untouched 3000-line test file solves half the problem. Mirror the package, so logic/comments.py pairs with tests/logic/test_comments.py:

uv run --no-project python $S/split_module.py <test file> tests_layout.json \
    --package-dir <tests>/logic --init-doc "Unit tests for the logic package."

Assign whole test classes. Carving up a class stops being a pure move.

5. Prove it is a pure move

git show HEAD:<module> > /tmp/before.py
uv run --no-project python $S/verify_pure_move.py /tmp/before.py <package>

It compares every top-level definition before and after, ignoring the module qualification and relative-import depth the split introduces, and reports anything missing, unexpected, duplicated with drift, or changed. This is what makes a 50-file diff reviewable: "every definition is identical" is checkable, unlike the diff. Always re-derive the before side from git, never from your own earlier output. Re-run it after any cleanup pass.

For a test split, pass --strip-package <package> so the source package's module names are normalized too.

6. Then the suite

hogli test <tests>/

Static checks cover import wiring and equivalence, not behavior. Deferred imports inside function bodies fail at call time, so only running the tests catches a mistake there.

Traps worth knowing

• A re-export shim in __init__.py gives every symbol two bindings, so patching one leaves internal callers on the real implementation: the patch applies, the test passes, and nothing intercepted. .semgrep/rules/devex/no-init-reexports.yaml already blocks eager re-exports repo-wide; the silent-mock failure is the extra reason not to add one.
• Moving code makes its old semgrep findings new. Semgrep skips its baseline comparison for findings in files that did not exist in the baseline commit, so every pre-existing violation in the moved code comes back as blocking. Check before pushing: uv run --no-project --with semgrep semgrep --config .semgrep/rules/devex <package>/.
• Relative imports break one level down. The script deepens them, including inside function bodies. A missed one is invisible until the function runs.
• A models.py split is the one case that needs re-exports. Django imports an app's models package but does not recurse into it, so model classes in submodules never reach the app registry and every from ..models import Model caller breaks. Add aggregation imports to models/__init__.py; no-init-reexports.yaml exempts **/models/** for exactly this reason. The script warns when it writes a models/ package.
• A top-level statement after the first definition stops the split. A module-level if, an assert, or a registration call cannot be attributed to a concern, and copying it into every module would run its side effect once per module. Move it into a function or above the definitions first.

Left for you

• Decide on the section dividers the move leaves behind.
• Rename locals that shadow a new module name.
• Update any doc or skill that links the old path; a split leaves dead links behind.