# Refactor Subsystem > Execute a large-subsystem refactor (split a bloated module, extract a service, reorganize a domain) using an orchestrator/scout pattern driven by a spec file in `ai-docs/specs/`. Operates in seven phases — Inventory, Characterize+Extract, Plan, Approve, Execute, Verify, Crystallize — with mandatory three-output scouts (primary brief + findings.md + extracted-behaviors.md), spec-first enforcement via `scripts/specs.py`, ledger updates via `scripts/ledger.py`, and a conservative default that unknown code STAYS until explicit human approval. Refactors are behavior-preserving; bug fixes surfaced along the way become separate commits. - Skill: `khurrummahmood/refactor-subsystem` (Agent Skill, multi-file: 8 files) - Install (CLI): `npx skillmds@latest add khurrummahmood/refactor-subsystem` - Raw SKILL.md: https://api.skillmd.com/api/skills/khurrummahmood/refactor-subsystem/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: Coding & Dev Tools - Author: KhurrumMahmood (https://skillmd.com/u/khurrummahmood) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/khurrummahmood/refactor-subsystem --- # /refactor-subsystem > **Legacy code is a specification written in the wrong language. The first > refactor of a subsystem is not a refactor — it's an excavation.** You are the **orchestrator** for a multi-file, multi-commit structural refactor. Unlike `/fix-workflow` — which handles a single cluster of duplication or dead code — this skill produces the spec-driven, scout-fanned, human-approved split of a bloated module or subsystem. **How success is judged.** These are the gates the run will face — optimize for them, not for speed: - Characterization suite green at HEAD, after every batch, and at 6.1. - Per-batch coverage-path proof (R36): a named suite imports/patches each batch's destination modules — in the plan, re-checked per batch. - `phase-6-boundary.md` clean — or each finding explicitly waived in the Phase 4 sign-off block. - Sign-off scope honored token-for-token; nothing executed beyond it. - Every knowledge pointer resolves to real content. If a pointer dangles, STOP and flag it — do not improvise a recipe. Procedural detail lives in five knowledge files (plus one shared rubric in `_common/`): - `knowledge/operations.md` — worktree paths, scripts, venv conventions, cleanliness guard, archaeology recipe + report schema, test matrix, report directory layout. Read at the start of Phase 1. - `knowledge/execution-playbook.md` — Phase 5 batch execution protocol, two-commit discipline, micro-fix swarm dispatch, convention enforcement decision loop, caller-update wave. Read at Phase 5 start. - `knowledge/solid-gate-tests.md` — pass/fail rubric for the three-level SOLID quality gate plus the Phase 1.2.5 worked example. Read only when running Phase 6.3 (dispatched to sub-agent, not by orchestrator). - `knowledge/bootstrap.md` — Phase 0 stub-scaffolding playbook. Read only when `specs.py show` exits non-zero. - `knowledge/learnings.md` — 44 rules (R1–R44) distilled from prior refactors. Read on ambiguity; don't front-load. An L-number index at the bottom maps shakedown lessons to R-numbers. - `.claude/skills/_common/interface-depth.md` — shared rubric for deletion test, caller-knowledge hiding, test surface, and adapter reality. Read when Phase 3 proposes new public modules/services or Phase 6 reviews the result. Two scout brief templates live in `agents/`: - `agents/inventory-scout.md` — Phase 1.3 template with `{{placeholder}}` slots the orchestrator fills in. - `agents/micro-fix-scout.md` — Phase 5.3.5 swarm template. ## Core beliefs 1. **The spec is the plan.** If the spec says `[x] IM-N`, the code must reflect it. If it says `[ ]`, the work hasn't happened. Drift between the two blocks commits (`scripts/specs.py coverage ` is the gate). 2. **Bidirectional retrofit.** The first pass against legacy code is not a one-way transformation. You read code, extract behavior that was never written down, feed it back into the spec, AND THEN refactor. The spec grows during the run. **Bootstrap corollary:** legacy code with no spec is the canonical case — not an error state. Scaffold the stub in Phase 0. 3. **Conservative default for unknowns.** If you read a function and cannot explain why it exists — via a spec item, an `extracted-behaviors.md` entry, or git archaeology — it **stays**. Deletion requires explicit human approval at Phase 4. 4. **Behavior preservation ≠ correctness.** A correct refactor of broken code is still a correct refactor. Never bundle a bug fix into the structural commit. Two-commit discipline. 5. **Scouts produce THREE outputs, not one.** Every scout returns (a) the primary brief, (b) `findings.md`, (c) `extracted-behaviors.md`. Scouts that return only the primary brief are incomplete and must be rerun. 6. **Read toward the standard, not the neighbor.** Conventions live in CLAUDE.md's Canonical Patterns section and in `.claude/docs/*.md` loaded per subsystem. Scouts extract conventions alongside behavior AND flag where the subsystem fails them. Gaps are findings. 7. **Deepen, don't just rearrange.** A split or service extraction only earns its keep when callers learn less and behavior concentrates behind a stable interface. Use the shared interface-depth rubric to reject pass-through modules and hypothetical seams. 8. **The ownership boundary is the target.** A refactor is not done because the old file went quiet. After a split, scan the whole new family: old shim, new package files, sibling modules with the same ownership prefix, URL-registered prototype routes, templates/JS loaded by those routes, and newly-created services. Boundary clean beats filename clean. 9. **For omnibus views, ownership beats file count.** Extract one behavior-backed responsibility at a time into a named service, keep public HTTP contracts stable, add a negative guard that prevents the responsibility from returning to the view, and only then decide whether package/file splitting is still useful. ## Scope See `knowledge/operations.md` for exact paths, venv rules, and the cleanliness guard. Summary: - **Worktree:** run wherever invoked. Confirm with `git rev-parse --show-toplevel` before Phase 1. - **Python:** `.venv/bin/python` for Django; plain `python3` for `scripts/specs.py`, `scripts/ledger.py`, `scripts/chunk_file.py` (all stdlib-only). - **Cleanliness guard:** `code_roots` must be clean (no unrelated uncommitted edits) before Phase 1 AND before every Phase 5 batch. Commands in `knowledge/operations.md`. - **Argument:** a spec id that resolves to `ai-docs/specs/.md`. Validate with `python3 scripts/specs.py show `. If the spec doesn't exist, run Phase 0 — do not abort. ## Resuming mid-refactor A refactor legitimately spans sessions (R33: two preparation sessions plus one execution session is the healthy shape). On entering with a fresh context mid-run: 1. **Infer the current phase:** `python3 scripts/specs.py coverage ` plus `ls reports/refactor//` — the artifacts present (see `knowledge/operations.md` "Report directory layout") tell you which phase last completed. 2. **Re-read that phase's knowledge file.** Phase 5 means `knowledge/execution-playbook.md` IN FULL — not from memory. 3. **Restate the approved scope and waivers** from `phase-3-plan.md` §Sign-off before any edit — and end the restatement by re-arming beliefs 3 and 4: unknown code STAYS; deletion needs recorded human approval. (One step: a scope restatement that omits the belief re-arm is incomplete.) 4. **Re-run the cleanliness guard** (commands in `knowledge/operations.md`). Do not resume a Phase 5 batch with the playbook unread or the sign-off scope unloaded. ## Mode detection Determined at the start of Phase 1 via the SOLID audit (§1.2.5): ### Standard mode (service extraction) **Trigger:** `code_roots` names multiple files, OR a single file whose SOLID audit shows 0–2 SRP "and"s. **Primary axis:** IM item → service extraction. **Plan shape:** `IM-N: extract X into service Y`. ### Decomposition mode (domain splitting) **Trigger:** `code_roots` names a single file ≥ 2,000 LOC with 3+ SRP "and"s. **Primary axis:** responsibility cluster → new file. **Plan shape:** `Domain cluster D → new file /.py`. **What changes:** Phase 1 adds a SOLID audit (§1.2.5), Phase 3 organizes the plan by domain cluster (§3.2.1), Phase 5 adds a caller-update wave (execution-playbook §5.6). The 7-phase workflow and safety net are unchanged. **Edge case:** multi-file `code_roots` where ONE file dominates (e.g., 10K LOC + 4 × 200 LOC satellites). Run the SOLID audit on the dominant file. If it scores 3+ "and"s, decomposition mode applies to that file; the satellites go through standard mode as part of the same spec. ### Packaging mode (flat-cluster consolidation) **Trigger:** `code_roots` names a flat cluster of ≥3 sibling files sharing a common prefix (e.g. `app/api/settings_*.py`, `app/pages/site_config_*.py`), where the work is *topology rearrangement* — pack siblings into a folder named for the prefix and strip the prefix from each filename — not service extraction or domain splitting. ADR 0006 (folder-organization ≥3-siblings) is the authorizing decision; ADR 0010 (pages-mirror-routes) adds a route-segment constraint when the cluster lives under `app/pages/`. **Primary axis:** Cluster prefix → folder name. Per-file prefix stripped on the move so the folder + filename together reconstruct the old name without duplication (`api/settings_global.py` → `api/settings/global.py`, NOT `api/settings/settings_global.py`). **Plan shape:** `IM-N: package _*.py into /`. Each IM item lists the move table: `(old_path → new_path, callers to update, templates to update)`. **What changes:** - Phase 1 SOLID audit is **skipped** — no SRP work; the cluster's responsibilities don't change, only their location. - Phase 1 inventory becomes a **move table**: every file in the cluster, every import statement that references it, every template `{% include %}` and `template_name=` that names it, every URL pattern that imports its view. The scout's job is to enumerate, not classify. - Phase 3 plan organizes IMs by cluster (one IM per prefix, not per file). - Phase 5 batches one cluster at a time. For each cluster: (1) `git mv` each file with prefix-stripped destination, (2) update every import found in the move table, (3) update every template/URL reference, (4) add `__init__.py` re-exports if any caller imports from the parent (`from api import settings_global` → must keep working via `__init__.py`, OR every such caller updates in the same commit), (5) run the always-suite + targeted backend suite. - For Markdown-heavy path references, docs, repo-layout moves, or any packaging batch where deterministic path rewriting is the main risk, use `/move-path` to dry-run and apply the filesystem/reference portion instead of hand-maintaining relative links. Keep behavior-preserving characterization and import-contract work in this skill. - Phase 6 verification: the topology-drift detector reports zero findings under the cluster's new path; pre-commit lint registry resolves cleanly with `--all`; one Playwright pass on any UI route whose template references moved. **Edge cases:** - **Mixed cluster** — most files share a prefix but 1–2 don't. Either (a) include the outliers if a prefix-rename would naturally fit, or (b) leave them flat and document why in the spec's §Exceptions. Don't force-fit. - **ADR 0010 route-mirror constraint** — pages clusters get an extra rule: the parent folder must mirror a URL route segment. So `app/pages/sites/wizard.py` mirrors `/sites/.../wizard/`. The packaging move and route-mirror placement happen together; the `find-folder-topology-drift` `pages_route_mirror` band catches filenames that forgot to drop the parent prefix. - **Wire identifiers (R44)** — packaging IS a rename. Apply the three-bucket triad: Python imports follow the move; wire identifiers (URL `app_name`, `template_name=` strings, FK string literals if any) stay frozen unless explicitly migrating; doc prose follows. - **Mixed-mode spec** — a single spec can carry packaging IMs alongside decomposition IMs (e.g. phase-2: pack `api/settings/` AND decompose `services/ai_sidecar/`). Run packaging IMs first — they're the lower-risk cleanup, and a clean cluster makes decomposition planning easier. ## Phase 0 — Bootstrap (only when the spec does not exist) Run `python3 scripts/specs.py show ` at the top of Phase 1. If it exits 0, skip Phase 0 entirely. If it exits 1 ("no spec with id"), read `knowledge/bootstrap.md` and follow steps 0.1–0.4 to scaffold the stub, commit it, and then start Phase 1. Any other non-zero exit is an abort signal — report and stop. Key invariants (no need to load the playbook to remember these): - The scaffold locks in `code_roots` — confirm scope with the human BEFORE running `specs.py init`. - A stub spec has `status: STUB`; Phase 1.1.5's inventory gate will recognize that and allow scouts to populate the narrative. - Commit the stub as a single-file commit before any Phase 1 work. Where committing is not permitted (review-only runs, CI sandboxes), record the intent in the run report and defer the commit to the first permitted moment; downstream phases run against the on-disk spec normally. ## Phase 1 — Inventory Goal: know every file, symbol, public surface, and import edge in the subsystem. No interpretation yet. ### 1.1 Load spec and ledger ```bash python3 scripts/specs.py show python3 scripts/specs.py coverage python3 scripts/ledger.py list --decision split_queued,monitor ``` Record: `code_roots`, current `[ ]` / `[~]` / `[x]` states, ledger entries for files in `code_roots`. (`ledger.py list` exits 1 with "no entries match" when nothing matches — a normal empty result, not a failure.) Verify the venv before Django commands (fall back to `$PYTHON_VENV_PATH` if the worktree lacks its own `.venv`): ```bash if [ -z "${PYTHON_VENV_PATH:-}" ] && [ ! -x .venv/bin/python ]; then echo "ERROR: no venv. Install dependencies per CLAUDE.md." exit 1 fi ``` The guard applies only when the current phase will issue Django/`manage.py` commands; phases that issue none (e.g. Phase 0–1 inventory work) note the missing venv and proceed. If `coverage` reports drift (checkmark lag or orphan refs), **fix the drift first** — either as a sub-task or abort and report. A spec that already drifts is not a safe refactor target. ### 1.1.5 Inventory gate — verify the spec matches reality (mandatory) The spec's narrative inventory can lag actual file contents. Running scouts against a stale narrative silently orphans load-bearing code (R14). ```bash python3 scripts/specs.py inventory-check ``` Gate outcomes: - **Clean** — counts and names match. Proceed to 1.2. - **Drift ≤ 10%** — minor. Log the delta in `reports/refactor//phase-1-inventory-gate.md` and add missing symbols to the chunking plan. No spec edit required. - **Drift > 10%** OR spec is `status: draft` / has stub-inventory warning — **pause and update the spec first**. New symbols become new or expanded IM items in `ai-docs/specs/.md`. Re-run until clean. - **STUB (from Phase 0 scaffold)** — expected. Scouts populate the narrative at Phase 1.3; stub warnings get removed at Phase 2b consolidation. Not a blocker. - **Orphan regions** (contiguous spans unmentioned by the spec) get dedicated "orphan chunks" in Phase 1.3.0 — higher-ROI scout targets than spec-planned chunks. ### 1.2 Determine the relevant convention docs Map `code_roots` to the right convention sources so every scout flags the same violations with the same naming: 1. **Always in scope:** `.claude/CLAUDE.md` Canonical Patterns section and any `ai-docs/specs/.md` whose `code_roots` overlap. 2. **Conditionally in scope** — pick from the "Supplementary Documentation" table in `.claude/CLAUDE.md`. The docs live under `.claude/docs/`, so write the full path in `convention-sources.md`. - Crawling / sitemaps / downloads → `.claude/docs/pipelines.md` - Extraction / AI training → `.claude/docs/known-issues.md` - PTID work → `.claude/docs/ptid-pipeline.md` - Custom-site imports → `.claude/docs/custom-site-data.md` - Models / views / services / tasks structure → `.claude/docs/architecture.md` - Deployment / env vars → `.claude/docs/configuration.md` 3. **Out of scope:** any doc whose trigger doesn't match. Context is expensive. **Absence fallback.** If the named convention sources do not exist in the host (no `.claude/CLAUDE.md`, no `.claude/docs/`), substitute a generic language/framework-hygiene rule table, record the substitution in `convention-sources.md` for Phase 4 human audit, and do NOT import the worked example's helper names below as rules. Write the resolved list to `reports/refactor//convention-sources.md`. Every scout brief references this file. **Scope each rule with a path predicate.** A convention extracted from `docs/pipelines.md` that only applies to crawl tasks must not be flagged as a violation in a view or service file. For every convention entry, record: ```markdown | Rule short | Canonical helper | Anti-pattern regex | Applies when path matches | |---|---|---|---| | AR-safe-dispatch | TaskDispatchService.safe_dispatch | `\.delay\(|\.apply_async\(` | `core/(views|tasks|services)/.*\.py` | | AR-ensure-site | SiteConfig.ensure_for_site | `get_or_create\(site=` | `core/.*\.py` (any file that imports SiteConfig) | | AR-safe-int-user-input | core.input_utils.safe_int | bare `int\(request\.(POST|GET)` | `core/views.*\.py` (views only — not services, tasks, or management commands) | ``` Scouts must check the `Applies when` column before flagging a violation — a task file using `get_or_create` for a non-Site model is NOT a violation of AR-ensure-site. Bare application of "the convention doc says X" without the path predicate turns local norms into false global failures and buries real issues in noise. ### 1.2.5 SOLID audit (decomposition mode — mandatory when triggered) **Skip in standard mode.** When decomposition mode is active, run this audit BEFORE chunking. It produces the responsibility clusters that guide chunking, scout briefs, and the Phase 3 split plan. **Step 1: SRP sentence test.** Describe the file in one sentence: "This file handles X and Y and Z." Count the "and"s: - 0 → cohesive. Switch to standard mode. - 1–2 → check whether responsibilities are facets of one job or genuinely separable domains. Facets = standard mode. - 3+ → decomposition mode. Each "and" clause maps to a cluster. Evaluation rule: "and"s connecting facets of one domain count as 0; "and"s connecting independently-understandable domains count as 1 each. See `knowledge/solid-gate-tests.md` for worked examples. **Step 2: Responsibility cluster mapping.** Group every top-level function/class by domain. Use `scripts/chunk_file.py --format markdown`: ```markdown | Cluster | Functions | LOC | Target file | |---|---|---|---| | Import & Validation | import_products_task, ... | 171 | _import.py | | Crawling | bulk_crawl_task, ... | 1983 | _crawling.py | ``` Clusters under ~100 LOC are merge candidates. **Step 3: Intra-file DRY scan.** Use structural AST comparison (R27). `scripts/specs.py solid` (Gate 2) normalizes `ast.dump()` output. At this stage consume ONLY the Gate-2 (DRY) section of its output: L1 checks for `phase-1-solid-audit.md` — the very file this step is producing — so an L1 SKIP/FAIL and a non-zero overall exit are expected here and are not abort signals. Look for identical try/except shapes, duplicated setup sequences, multiple implementations of the same abstraction. Record each as a cross-cutting concern: ```markdown ### Cross-cutting concerns 1. **Task lifecycle** — 94 identical try/except blocks → @task_lifecycle decorator 2. **Proxy setup** — 11 duplicated sequences → service consolidation 3. **Progress tracking** — 2 incompatible systems → unified abstraction ``` Cross-cutting concerns become **Batch 1** of Phase 5 — consolidate BEFORE splitting by domain (R26). **Step 4: Linear flow test.** For 3 representative functions (small, medium, large), trace the execution path. Functions that call helpers thousands of lines away (past unrelated clusters) fail linearity — the helper should move with its caller. Write the audit to `reports/refactor//phase-1-solid-audit.md`. It feeds Phase 3's split plan. ### 1.3.0 Chunk oversized files (mandatory for files > 2,000 LOC) A single scout on a 10K-LOC file produces shallow output (R15). Every file in `code_roots` whose LOC exceeds 2,000 gets chunked. ```bash python3 scripts/chunk_file.py --token-budget 8000 --format json \ --output reports/refactor//inventory/__chunks.json python3 scripts/chunk_file.py --token-budget 8000 --format markdown \ --output reports/refactor//inventory/__chunks.md ``` Rules: - **Defaults:** `--token-budget 8000 --loc-budget 2500`. Tune only when scouts overflow or coordination breaks down. - **Two "orphan" notions — don't conflate them.** The chunker's "orphan regions" are coverage gaps between chunks; §1.1.5's orphans are spec-unmentioned spans. Inspect every chunker gap region before creating anything: blank/trivial separators (whitespace, lone comments) are folded into the adjacent chunk and their disposition recorded in the chunk map. Only substantive uncovered spans (real code the chunk map misses) and §1.1.5 spec-unmentioned spans become orphan chunks — `orphan-1`, `orphan-2`, ... — with their own scouts. Rank orphan IM proposals first in Phase 2.2 (R14; original lesson L-12). - **Spec-guided cleavages:** if the spec enumerates IM-group line boundaries, pass `--loc-hints ` to bias toward them. - **Files ≤ 2,000 LOC skip chunking.** One scout per file, basename-keyed outputs. - **Non-Python files:** chunker only handles Python. Flag in `__chunks.md` for manual planning if needed. - **Basename-qualify every chunk ID before dispatch (R35).** The chunker emits raw IDs `C-01`, `C-02`, `orphan-1`. Two chunked files in the same spec run (e.g., `tasks.py` and `services.py`) both produce `C-01`, which silently clobbers scout outputs and collides provisional-ID regex at Phase 2.2. Before writing the chunk map, the orchestrator rewrites every raw ID to `__`: `tasks__C-01`, `services__C-01`, `tasks__orphan-1`. Files that skip chunking get a single chunk id `__C-01` so the scheme is uniform. Output files, provisional item IDs, and the canonical short-code regex (R21) all use the qualified form. Write the chunk map to `reports/refactor//inventory/__chunks.md`. The orchestrator REWRITES that file — overwriting the chunker's markdown output at the same path — into the format below (basename-qualified IDs, archaeology-owner column); the chunker's raw output survives at `__chunks.json`. Dispatch scouts only from the rewritten map: raw chunker IDs like `C-01` are unqualified and would break R35 qualification. ```markdown # Chunk map — ( LOC total) | Chunk ID | Lines | LOC | ~Tokens | Declarations | Archaeology owner | |---|---|---|---|---|---| | tasks__C-01 | 1–1480 | 1480 | 7900 | imports, logger, ... (14 total) | orchestrator | | tasks__C-02 | 1481–2875 | 1395 | 7200 | bulk_crawl_sitemaps_task, ... (8 total) | orchestrator | | tasks__orphan-1 | 9240–10118 | 879 | 4600 | auto_generate_exports_task, ... (11 total) | orchestrator | ``` **Then write the dispatch manifest** the §1.3 subprocess loop reads: `reports/refactor//inventory/chunks.jsonl`, one JSON object per basename-qualified chunk across ALL chunked files (and the single `__C-01` chunk of each unchunked file): ```json {"chunk_id": "tasks__C-01", "file": "core/tasks.py", "line_start": 1, "line_end": 1480, "declarations": "imports, logger, ... (14 total)", "archaeology_owner": "orchestrator"} ``` `chunk_file.py` does NOT emit this file — its JSON output is one per-file object with raw chunk IDs (`chunks[].id`) and no `archaeology_owner`. The orchestrator builds `chunks.jsonl` from the chunk map in the same pass that basename-qualifies the IDs (R35) and assigns archaeology owners. ### 1.3 Dispatch the inventory scouts (parallel) For each chunk (or each small file that skipped chunking), dispatch one `general-purpose` sub-agent (a read-only agent type such as `Explore` cannot satisfy the three-file output contract). Scouts run in parallel — one message, N tool calls. **Use `agents/inventory-scout.md` as the brief template.** Substitute every `{{placeholder}}` with the chunk's values (file path, line range, chunk id, spec id, archaeology owner, worktree, venv path, declarations from the chunker). Do not summarize the template. Archaeology: if the chunk map marks the archaeology owner as "scout", the brief tells the scout to run `git log --follow` on its range. If marked "orchestrator", the orchestrator handles Phase 1.4 for that file in parallel with scout dispatch. (Ownership split by churn — L-7.) #### Dispatch mode — Agent tool vs subprocess The `Agent` tool works only one level deep. If `/refactor-subsystem` is itself invoked as a sub-agent (e.g. a bigger workflow spawns it), the orchestrator has no `Agent` tool to fan out with and silently collapses to single-threaded inventory — a bad outcome on a 10K-LOC target. For nesting-safe fan-out, dispatch each chunk as a `claude -p` subprocess via `.claude/skills/_common/dispatch_scout.sh`. Each subprocess is a brand-new Claude Code process with the full tool set, so this works at any nesting depth. ```bash # One subprocess per chunk; parallelize with `&` + wait. The scout # writes its three output files itself (primary brief, findings, # extracted); dispatch_scout.sh verifies the primary brief path. while read -r chunk; do cid=$(jq -r '.chunk_id' <<<"$chunk") file=$(jq -r '.file' <<<"$chunk") ls=$(jq -r '.line_start' <<<"$chunk") le=$(jq -r '.line_end' <<<"$chunk") basename=$(basename "${file%.*}") out="reports/refactor/${SPEC_ID}/inventory/${cid}__L${ls}-L${le}.md" .claude/skills/_common/dispatch_scout.sh \ .claude/skills/refactor-subsystem/agents/inventory-scout.md \ "$out" \ file="$file" line_start="$ls" line_end="$le" chunk_id="$cid" \ spec_id="$SPEC_ID" basename="$basename" \ declarations="$(jq -r '.declarations' <<<"$chunk")" \ archaeology_owner="$(jq -r '.archaeology_owner' <<<"$chunk")" \ worktree="$(git rev-parse --show-toplevel)" \ venv=".venv/bin/python" \ branch="$(git branch --show-current)" & done < "reports/refactor/${SPEC_ID}/inventory/chunks.jsonl" wait ``` **Tradeoffs.** Subprocess dispatch adds ~4–8s spawn + full context reload per scout vs ~0s for `Agent`. Use `Agent` when the skill runs at the top level (cheaper, no spawn overhead). Use subprocess dispatch when the skill may be invoked nested, or when scout context isolation is worth more than the spawn cost. The same subprocess pattern works for Phase 5.3.5's micro-fix swarm when it runs nested — dispatch `agents/micro-fix-scout.md` through `dispatch_scout.sh` the same way. The swarm's dispatch protocol and guardrails (edit-only sub-agents, serial orchestrator commits) are in `knowledge/execution-playbook.md` §5.3.5; there is no separate subprocess wrapper script for it. ### 1.4 Git archaeology (trigger-based, NOT optional) See `knowledge/operations.md` for the full recipe. Summary: - **≤ 500 LOC AND ≤ 20 commits** → scout runs it inline. - **Everything else** → orchestrator runs it in parallel with scouts. - **≥ 50 commits** → archaeology is **mandatory** (R17). The archaeology file must include at least 3 load-bearing LR-T candidates with `` tags. The recipe uses a subject-word filter (`fix|retry|timeout|crash` — known terms; the host project extends the list, see the host-adapter slot in `knowledge/operations.md`) to find high-signal commits. Record findings in `reports/refactor//archaeology/.md` per the schema in `knowledge/operations.md`. ### 1.5 Consolidate the inventory Write `reports/refactor//phase-1-inventory.md`: - File-by-file table: path, LOC, symbol count, public imports, outbound imports. - Chunk table for every chunked file: chunk id, line range, LOC, declaration count, scout status, archaeology owner. - Dependency graph (ASCII or DOT). - Hot-spot list (functions/classes above complexity thresholds). - Counts: total LOC in scope, public surface, chunks dispatched, orphan chunks, inventory-gate delta. **Gate for Phase 2** (chunk-level, not file-level — R2): 1. **Coverage** — `sum(end - start + 1) == LOC` for every chunked file. 2. **Three outputs per chunk** — all three scout files on disk. 3. **Archaeology present where required** — every ≥ 50-commit file has ≥ 3 LR-T candidates; every ≤ 500 LOC / ≤ 20 commits file has inline archaeology or a note. Files in neither bracket (e.g. a large file with modest history) get orchestrator-owned archaeology per operations.md's "everything else" rule, producing a tagged report — held to the ≥ 3 LR-T standard where history supports it, else the report records the shortfall and the reason. 4. **No empty primary briefs** — silent-fail signal. Re-dispatch. Missing outputs → re-dispatch. Do not proceed until all four conditions hold for every chunk. ## Phase 2 — Characterize + Extract (parallel) Goal: freeze current behavior with tests, AND pull the "why" out of code into the spec. These run in parallel and inform each other. ### 2.1 Characterization tests Write temporary tests in `tests/test__characterization.py` that capture the **current** behavior of the public surface. They must pass against HEAD. Per public function / class entry point: - **Import-level safety tests** — `import core.tasks; assertTrue(hasattr(core.tasks, 'foo_task'))` for every public name. Cheap, catches "forgot to re-export." - **Behavior snapshots** for non-trivial logic: fixture in, expected dict out. Golden files in `snapshots/` for anything > 10 keys. - **Skip private helpers** — they move with callers. - Mark the file with `# spec:::characterization` — transient, deleted in Phase 7. **Decomposition-mode characterization pins *structure*, not *behavior* (L-44).** The right test shape is a `TaskImportabilityTest` (every public symbol importable from the original path), `TaskSignatureTest` (function signatures unchanged), and `TaskRegistrationTest` (Celery tasks still registered with their original names + options). Behavior tests are the domain test suites' job. Structure pinning is sufficient ONLY with the per-batch coverage-path proof (plan item 7, R36): never trust green from a suite with no path into the moved code. **Shim compatibility is mandatory for Django module splits.** When an old view/task/service module becomes a package or re-export shim, add characterization tests that pin: - imports from the old module path (`core.views.site_config`, `core.views.brand_downloads`, etc.), - imports from the parent package (`core.views` when it re-exports view classes), - the old URL names and view callables that remain registered. Move callers only when there is a behavior reason. Preserving imports first keeps large splits reviewable and lets tests fail at the public contract instead of at random import sites. Run the tests and confirm they pass on HEAD: ```bash .venv/bin/python manage.py test tests.test__characterization \ --settings=app.settings_test_sqlite -v 2 ``` If any fail on HEAD, either the test is wrong (fix it) or the current behavior is already broken (flag P0, do NOT adjust the test). ### 2.2 Extraction pass Read `reports/refactor//extracted/*.md` (all scout outputs) and consolidate into `reports/refactor//extracted-behaviors.md`. Each scout output is named `{chunk_id}__L{start}-L{end}.md` (see `knowledge/operations.md` "Report directory layout" and the completeness contract in `agents/inventory-scout.md`). Missing or mis-named files indicate an incomplete scout — re-dispatch rather than proceeding. **Provisional-to-canonical ID reassignment (R16).** Scouts propose IDs with basename-qualified chunk-id prefixes (`tasks__C-03-EX-2`, `services__orphan-1-IM-1`). The consolidation pass reassigns surviving entries to canonical IDs by incrementing the highest existing number in the spec's AR/EX/IM/LR-T sections. The basename qualifier in every prefix (R35) guarantees no collision across parallel scouts even when two different chunked files share a raw chunk number. **Merge by summary before assigning canonical IDs.** Every extracted entry carries a one-line purpose summary. Merge semantically-identical entries across chunks BEFORE reassigning IDs — two scouts proposing `*-AR-N: safe_dispatch catches all exceptions` from different call sites merge into one canonical entry with a `**Seen in chunks:**` line. Rank orphan-chunk IM proposals first (R14). Orphan candidates were drifting out of spec — higher extraction ROI than spec-planned ones. Consolidated file structure: ```markdown # Extracted behaviors — ## IM candidates (behaviors worth a new IM item) ## AR candidates (structural constraints worth an AR item) ## EX candidates (non-obvious rules — gotchas) ## LR-T candidates (technical lessons — "why" behind defensive blocks) ## Remove candidates (appear dead. STAY until Phase 4 sign-off.) ## Investigate (unclear semantics. Default if in doubt.) ``` **The extraction pass is the most important step.** If rushed, the refactor silently destroys load-bearing code. For a 10K-LOC subsystem, Phase 2.2 takes longer than Phase 5 (R12). ### 2.3 Findings consolidation Read `reports/refactor//findings/*.md` from all scouts and consolidate into `reports/refactor//findings.md`. Filename convention matches §2.2 — `{chunk_id}__L{start}-L{end}.md` per the inventory-scout completeness contract. Two sub-steps: **2.3.1 Cross-scout dedup (R18)** — run BEFORE tiering. Parallel scouts overlap when one chunk cross-references call sites in another. Build a `(file, line-range, convention-violated)` fingerprint and merge duplicates: ```markdown ## P2: bare task.delay() at views_training.py:526 **File:** core/views_training.py:526 **Reported by:** chunk T-6 (tasks.py range), chunk D-1 (task_dispatch.py range) **Observation:** `training_task.delay(site_id)` — bypasses safe_dispatch **Convention violated:** AR-2 (TaskDispatchService.safe_dispatch) **Why it matters:** No Celery retry; silent failure under broker outage **Recommended disposition:** fix ``` Rules: - **Strongest disposition wins** on disagreement; note the disagreement. - **List all source scouts** in `**Reported by:**` — the micro-fix swarm uses this as a provenance trail. **2.3.2 Tier the deduplicated findings** into P0→P3: ```markdown ## P0 — blocks the refactor ## P1 — must be addressed this cycle (separate commits) ## P2 — should be addressed soon, not this cycle ## P3 — nice to have, parking lot ## Convention adoption rates (from `specs.py violations `) | Convention | Canonical | Compliant | Violating | Compliance % | Top offenders | ``` Run `python3 scripts/specs.py violations ` to populate the Convention Adoption table (R13). The full violation list is repo-wide; Phase 5.4 filters it to `code_roots` unless whole-repo enforcement was approved at Phase 4. ## Phase 3 — Plan Goal: concrete split plan, informed by inventory + extracted behaviors + findings. ### 3.1 Update the spec with extracted behaviors **Spec-first enforcement.** New IM / AR / EX / LR-T items from `extracted-behaviors.md` get added to `ai-docs/specs/.md` **before** any code moves. Enter as `[ ]` (or `[x]` for AR/EX that document pre-existing decisions with no code work needed). Each new item gets a unique incremented ID — do not reuse. Run coverage to confirm the spec is still parseable: ```bash python3 scripts/specs.py show python3 scripts/specs.py coverage ``` `documented_only` / `checkmark_lag` growth is expected at this phase (new items have no refs yet). `is_clean: false` is fine here. ### 3.2 Write the split plan `reports/refactor//phase-3-plan.md` contains: 1. **Target file tree after the refactor** — every new file, every survivor, every deletion. 2. **Symbol → destination map** — every public symbol mapped to its new home, with its `# spec:::IM-N` comment. 3. **Shim strategy** — re-export shim with enumerated re-export lines, or caller updates required. 4. **Endpoint contract matrix** for view/API modules — every endpoint in scope classified by route name/path, auth level (anonymous/user/staff), method, CSRF expectation, side effects, response shape, and external boundary (network, credentials, command execution, filesystem, Celery). 5. **Parallel renderer matrix** — every renderer/presenter of the same concept. Examples: dashboard rows, polling JSON, sidebar statuses, prototype rows, settings page forms, and admin diagnostic panes. The plan must name the canonical producer before moving presentation logic into services. 6. **Batch plan** — each batch leaves the repo green and is individually revertable. Rule of thumb: one `[batch-tag]` prefix per batch, passes its test scope independently. For 10K LOC, expect 5–15 batches. 7. **Test strategy per batch** — which modules' tests need to pass, WITH grep evidence that at least one named suite imports or patches each batch's destination modules (R36 generalized to batch level). The plan shows the grep output; the Phase 4 reviewer approves coverage, not suite names. A batch whose grep comes back empty has no test strategy yet — fix that before Phase 4. 8. **Interface depth checks** — for every new public service/module, shared helper, or adapter seam, include the compact section from `.claude/skills/_common/interface-depth.md`: deletion test, caller knowledge removed, test surface, adapter count, and decision. 9. **Guard strategy** — negative test or lint proposal that prevents the extracted responsibility from returning to the old layer. Guard tests and lint guards are peers; choose the cheaper shape that protects the invariant. 10. **Rollback plan** — revert the batch, keep characterization tests, re-plan. 11. **Risks** — concrete failure modes with mitigations. ### 3.2.1 Decomposition-mode plan structure **Skip in standard mode.** When decomposition mode is active, the Phase 3 plan uses a different primary axis. **Primary axis: domain cluster → new file** (from the SOLID audit), not IM item → service. ```markdown ## Target file tree / __init__.py → re-export shim common.py → shared imports, constants, cross-cluster helpers crawling.py → Crawling cluster discovery.py → Discovery cluster extraction.py → Extraction & AI cluster ``` **Directory packages over flat naming (R29).** Prefer `/.py` over `_.py`. The `__init__.py` is the natural re-export shim; it matches Django conventions. Match existing flat naming for consistency if the codebase already uses it. **CRITICAL: File→directory migration is atomic.** Python cannot have both `tasks.py` and `tasks/` simultaneously (`import core.tasks` becomes ambiguous). The migration is a single commit: 1. `git mv core/tasks.py core/tasks_old.py` 2. `mkdir core/tasks/` + create `__init__.py` + create domain modules (read source from `tasks_old.py`) 3. `rm core/tasks_old.py` For modules with many callers, flat naming is lower-risk because it avoids the atomic rename. **Within each cluster:** - All functions/classes for the domain move as a unit. - Helpers called only by that domain move with them. - Shared helpers stay in `common.py`. - Service extraction happens within clusters, not across. **Cross-cutt …(truncated)