davidvujic Skill: polylith-migrate-dedupe
📐 Scope vs sibling skills. This skill is opportunistic deduplication that may be triggered any time during refactoring when duplication candidates surface. It is not the canonical place for the structural decompositions:
- For "split this big component into smaller ones", use
polylith-migrate-split-big-component(it already includes a dedup-analysis subsection — usually sufficient on a first migration).- For "this component's
core.pymixes domains", usepolylith-migrate-split-component-internals.- For "two projects have overlapping code, split shared from project-specific", use
polylith-migrate-isolate-shared-and-project-logic.Use
polylith-migrate-dedupewhen none of the above fits cleanly — e.g., duplication discovered across already-extracted components that don't map to a structural split.
Goal
Identify duplication candidates during the migration process and execute controlled deduplication for user-approved candidates.
When to Use
- After splitting the big component or extracting standalone modules.
- When potential duplication between components is suspected.
Classification
Use this table when deciding whether a candidate is a real duplicate:
| Class | Definition | Action |
|---|---|---|
| Identical | Same logic, same control flow, only trivial differences (variable names, formatting, ordering of independent statements). | Extract into a shared component. Both call sites import from it. |
| Similar | Same purpose, slightly different behaviour (e.g., different default arguments, project-specific fields on an otherwise shared model). | Extract a parameterized shared component. Project-specific behaviour passes in as arguments or subclass hooks. Avoid forcing a one-size-fits-all signature. |
| Coincidental | Looks similar (same function name, same shape) but serves unrelated purposes. | Leave alone. Sharing here would couple two domains that should evolve independently. |
When to parameterize vs. keep separate
- Parameterize when the core logic is identical and only data/config differs.
- Keep separate when control flow or structure diverges (different frameworks, different patterns) — forcing a shared abstraction here creates a brittle "shared core" that grows project-specific flags over time.
- Extract shared base + per-project wrappers when there's a significant shared core but non-trivial project-specific logic around it.
Worked example — logging
Two projects each had their own init_logging. The core (structlog setup, base log levels, JSON formatter) was identical; the differences were:
- Project A added loggers for
httpx,backoff. - Project B added a logger for
confluent_kafka_helpers.
The shared component exposed an init(config, *, extra_loggers=None, cache_logger_on_first_use=False) function. Each project's base calls init with its own extra_loggers dict. No coincidental coupling, no version skew, and adding a third project requires only its own dict — not a change to the shared component.
Shared-component naming
When creating a shared component to deduplicate code, name it after the domain or capability it represents, never after how it's used. Good: logging, kafka_client, merchant_serializer. Bad: shared_utils, common, helpers, misc. Generic-named bricks attract more code over time and become the next thing that needs decomposing.
Inputs
From migration/<PROJECT>/state.md:
TARGET_TOP_NS- Verification commands (
RUN_TEST_CMD,RUN_LINT_CMD,RUN_TYPECHECK_CMD).
From migration/<PROJECT>/manifest.md:
- Module map of components.
All inputs from
state.mdare assumed to satisfy the validation rules inpolylith-migrate-discover(### Validation rules). Validate before proceeding.
Steps
1. Identify Duplication Candidates
- Use
directory_treeandgrepto scan for overlapping logic between components. - Classify candidates by type:
- Identical: Code that is exactly the same.
- Similar: Code that serves the same purpose but with minor differences.
- Coincidental: Code that looks similar but serves unrelated purposes.
2. Present Candidates to the User
- Provide a list of duplication candidates, including:
- Component names.
- File paths.
- Type of duplication (identical, similar, coincidental).
- Risk assessment (low, medium, high).
- Ask the user to approve or reject each candidate for deduplication.
3. Execute Deduplication for Approved Candidates
- For each approved candidate:
- Identical Code: Extract the shared logic into a new component and update imports.
- Similar Code: Refactor to use shared logic or parameterize differences.
- Coincidental Code: Leave as-is.
- Update
pyproject.tomlto include any new components. - Run
POLY_CMD_PREFIX syncto synchronize the workspace.
4. Verify Changes
- Run
RUN_TEST_CMDto ensure no regressions. - Run
RUN_LINT_CMDandRUN_TYPECHECK_CMDif set. - Run
POLY_CMD_PREFIX checkto validate the workspace structure.
Verify
- All tests pass (
RUN_TEST_CMD). - Linting and type-checking pass (if set).
- The workspace structure is valid (
POLY_CMD_PREFIX check).
Common failure modes
| Symptom | Likely cause | Remediation |
|---|---|---|
Two pieces of code look identical but operate on different domains (e.g., both are validate(...) but one is for users, the other for transactions) |
Coincidental similarity, not real duplication. | Classify as "coincidental"; leave both in place. Resist the urge to share. |
The candidate shared component would pull in framework-specific dependencies (e.g., a "logging" shared brick that needs both confluent_kafka_helpers and httpx) |
Wrong shared abstraction — you're sharing the union of two project surfaces. | Revert and parameterize instead: keep the shared core minimal and pass project-specific values as arguments. See the "Pattern: Parameterize the shared component" guidance in polylith-migrate-split-big-component. |
Tests break after deduplication because mock.patch("<old.path>") no longer hits anything |
Patch strings reference the pre-dedup module path. | Update patch strings to the new shared module path. Validate by deliberately breaking the patched function and confirming the test fails. |
Done When
- Duplication candidates are identified and presented to the user.
- User-approved candidates are deduplicated.
- All tests and checks pass.
- The workspace structure is valid.
Commit
After verification passes, commit this phase to the migration branch:
git add -A && git commit -m "migrate(<PROJECT>): phase optional — dedupe"
Substitute <PROJECT> from state.md. This is an optional skill off the numbered main line, so the commit uses the literal phase optional label (no <N>). Do not proceed without a clean commit — the per-phase commit is the rollback point for the next phase's failure-mode tables.