Find Duplication

/find-duplication

Kotlin/JVM 2.4.10 branch

Trigger this branch only for manifest-selected authored .kt functions. Keep sibling _kotlin, read ../_kotlin/GUIDE.md, and enter through scripts/run_kotlin.py. It reports exact normalized direct-body token clones with package/signature/overload syntax. Matching tokens do not establish resolved calls, DSL or inline semantics, delegation, side effects, reflection, generated/Java/Gradle/runtime behavior, equivalence, or consolidation safety.

C# 14 / .NET 10 branch

Use scripts/run_csharp.py with the sibling _csharp provider; run it with --help for the exact CLI. It reports exact normalized direct method-body Roslyn token clones of at least five source lines from the current compiled manifest closure. Matching tokens are an advisory clone lead, not semantic or behavioral equivalence or evidence that consolidation is safe.

C++20 branch

Use scripts/run_cpp.py with the sibling _cpp provider; run the script with --help for the exact CLI. It reports exact normalized direct-body spelling clones from a current complete C++20 compiler-owned graph, retaining namespace, signature, and overload identity. Matching text is not semantic equivalence or evidence about ODR/ABI, specializations, dispatch, or external variants.

C17 branch

Use scripts/run_c.py with the sibling _c/c_lexical_facts.py provider; run python3 scripts/run_c.py --help for the exact CLI. This external-library branch reports exact normalized direct-body spelling as an advisory clone lead only; macro expansion, behavioral equivalence, and consolidation safety remain unresolved.

Dart v1

Dart v1 consumes the shared _dart D3 snapshot and reports exact normalized public-analyzer token clones for direct named functions or methods of at least five lines. It is a review lead only; behavioral equivalence and consolidation safety remain unproven.

SKILL_ROOT=".agents/skills/on-demand/find-duplication"
python3 "${SKILL_ROOT}/scripts/run_dart.py" \
  --project-root "$PWD" --target lib --facts /tmp/dart-d3-facts.json \
  --output-dir "$PWD/reports/duplication/dart"

Run the language branch that matches the target. Python, JavaScript, TypeScript, Go, and Java share a skill name and report vocabulary, but not a detector model or outcome claim.

Route before running

Inspect eligible source suffixes under --target:

• .py only: run the Python legacy triage branch.
• .ts/.tsx only: run the TypeScript lexical-evidence branch.
• .js/.jsx/.mjs/.cjs only: run the JavaScript lexical-evidence branch.
• .go only: run the Go exact-function evidence branch.
• .java only: run the Java exact-method evidence branch.
• .php only: run the PHP exact token-normalized named-body evidence branch.
• .rb only: run the Ruby exact Prism method-body spelling branch.
• .swift only: run the Swift exact normalized direct-body evidence branch.
• .rs only: run the Rust exact normalized function-body evidence branch.
• multiple supported families: run each branch into its own language report directory and summarize them separately. Do not merge their findings or apply one family's outcome contract to another family's evidence.
• neither: stop and report that this skill has no eligible source.

Use a host Python 3.11+ interpreter. The selected skill is self-contained: no repository-level scripts/, _common, toolkit virtualenv, or shared language adapter is part of either installed path.

For Rust, the exact installed closure additionally includes the sibling _rust/rust_lexical_facts.py. Rust v1 reports only named functions of at least five lines with identical whitespace-normalized lexical bodies. The result is a human-review lead, never proof that consolidation is safe; macros, cfg variants, traits/generics, closures, semantic equivalence, and runtime behavior remain unresolved.

SKILL_ROOT=".agents/skills/on-demand/find-duplication"
python3 "${SKILL_ROOT}/scripts/run_rust.py" \
  --project-root "$PWD" --target src \
  --output-dir "$PWD/reports/duplication/rust"

PHP, Ruby, and Swift are external-library branches. Load the selected skill with its sibling provider and read the matching on-demand guide:

• ../_php-project-lexical/GUIDE.md
• ../_ruby-project-lexical/GUIDE.md
• ../_swift-project-lexical/GUIDE.md

All three produce human-review clone leads only. The guides own the exact commands, normalization boundary, native gates, and non-claims; none proves behavioral equivalence or safe consolidation.

Python legacy triage branch

The Python branch preserves the original user journey: pinned lexical detection plus Python AST pattern detection, method-identity collapse, ranking, per-finding scout investigation, a dormant-code side-channel, and final triage.md/findings.json suitable for /fix-workflow handoff.

Python success contract

• Every final finding was present in ranked.json and has a valid scout JSON in scout/<finding_id>.json before it becomes actionable.
• classified.json preserves all scout verdicts and dormant candidates.
• triage.md and findings.json include the same fix_shape, notes, latent bug risk, and side-channel evidence.
• Production source is unchanged.

Python setup

PYTHON="${PYTHON:-python3}"
SKILL_ROOT="${SKILL_ROOT:-.claude/skills/find-duplication}"
SCAN_ID="scan-$(date -u +%Y%m%d-%H%M%S)"
REPORT_DIR="reports/duplication/${SCAN_ID}"
TARGET="src"
NPM_CACHE="${NPM_CACHE:-/tmp/engineering-skills-jscpd-cache}"
RUN_PY_JSCPD="$SKILL_ROOT"/scripts/run_jscpd_python.py
DETECT_PY="$SKILL_ROOT"/scripts/detect_python.py
COLLAPSE_PY="$SKILL_ROOT"/scripts/collapse.py
RANK="$SKILL_ROOT"/scripts/rank.py
REPORT="$SKILL_ROOT"/scripts/report.py
mkdir -p "$REPORT_DIR/jscpd" "$REPORT_DIR/scout"

Python Stage 1 — detect

Run the two family-local commands. They are independent and may run in parallel. --offline-ok preserves the legacy AST-only degraded mode when the exact jscpd cache is absent; the resulting report says skipped_lexical and must never be described as a clean lexical scan.

"$PYTHON" "$RUN_PY_JSCPD" \
  --target "$TARGET" \
  --output "$REPORT_DIR/jscpd" \
  --npm-cache "$NPM_CACHE" \
  --offline-ok

"$PYTHON" "$DETECT_PY" "$TARGET" \
  --project-root "$PWD" \
  --output "$REPORT_DIR/ast-findings.json"

The lexical wrapper pins jscpd@4.0.5, runs npx --offline, stages only eligible production .py files, and excludes tests, migrations, vendor, generated, report, output, and prior .jscpd-input trees. The AST detector is stdlib-only and retains the legacy categories: unsafe request integer parsing, shadow safe-conversion helpers, repeated LLM-call helpers, inline request-body JSON parsing, and same-name/same-arity cross-module candidates.

Python Stage 2 — collapse

"$PYTHON" "$COLLAPSE_PY" \
  --jscpd-report "$REPORT_DIR/jscpd/jscpd-report.json" \
  --ast-findings "$REPORT_DIR/ast-findings.json" \
  --target "$TARGET" \
  --project-root "$PWD" \
  --output "$REPORT_DIR/collapsed.json"

Expected stderr begins with [collapse]. Default filters remove tests, migrations, vendor/framework boilerplate, reports, and staging input. Python enclosing-symbol mapping uses stdlib ast inside the copied skill.

Python Stage 3 — rank

"$PYTHON" "$RANK" \
  --input "$REPORT_DIR/collapsed.json" \
  --output "$REPORT_DIR/ranked.json"

This preserves the original multiplicity × divergence × blast-radius ranking and P0/P1/P2 tiers.

Python Stage 4 — investigate

This is the only Python stage where LLM judgment runs. Investigate the top 10 ranked findings by default (or all when fewer exist). For each finding:

1. Expand agents/investigate.md with finding_id, the finding JSON, project_root, skill_root, and output_path.
2. Dispatch a fresh general-purpose sub-agent. Dispatch independent scouts in parallel when the host supports it.
3. Require the scout to read knowledge/false-positives.md, any host overlay, and knowledge/learnings.md when ambiguity matches a precedent.
4. Accept only schema-valid JSON using one documented fix_shape. Re-dispatch malformed output; never silently promote an unreviewed finding.

Merge the accepted scout files:

"$PYTHON" -c '
import glob, json, pathlib, sys
report = pathlib.Path(sys.argv[1])
out = {"findings": [], "dormant_candidates": []}
for name in sorted(glob.glob(str(report / "scout" / "*.json"))):
    data = json.loads(pathlib.Path(name).read_text())
    out["findings"].append(data)
    out["dormant_candidates"].extend(data.get("dormant_candidates") or [])
(report / "classified.json").write_text(json.dumps(out, indent=2) + "\n")
' "$REPORT_DIR"

Python Stage 5 — final report

"$PYTHON" "$REPORT" \
  --input "$REPORT_DIR/ranked.json" \
  --classified "$REPORT_DIR/classified.json" \
  --output-md "$REPORT_DIR/triage.md" \
  --output-json "$REPORT_DIR/findings.json" \
  --scan-id "$SCAN_ID"
ln -sfn "$SCAN_ID" reports/duplication/latest

The final report preserves the original scout fix_shape, notes, latent bug risk, /fix-workflow cluster:<finding_id> handoff, and dormant-code side-channel. Summarize counts by shape, the top three clusters, latent risks, the final artifact path, and the recommended next command in at most 10 lines.

TypeScript lexical-evidence branch

TypeScript v1 reports only lexical or near-lexical clone clusters where each complete site range fits one proven function declaration or block-bodied arrow symbol. It excludes generated, tests, declarations, vendor, dependencies, build, report, output, and staging paths. Distinct occurrences remain distinct; raw pairs cluster only through overlapping occurrences.

This branch has no TypeScript type checker, module resolution, React/Node framework model, caller proof, or refactor-safety claim. It does not run Python scouts or hand findings directly to /fix-workflow.

TypeScript setup and pipeline

Provision the exact cache deliberately outside the audit when needed:

NPM_CONFIG_CACHE="/path/to/jscpd-cache" npx --yes jscpd@4.0.5 --version

Then run all four installed stages:

PYTHON="${PYTHON:-python3}"
SKILL_ROOT="${SKILL_ROOT:-.claude/skills/find-duplication}"
SCAN_ID="scan-$(date -u +%Y%m%d-%H%M%S)"
REPORT_DIR="reports/duplication/${SCAN_ID}"
TARGET="src"
NPM_CACHE="${NPM_CACHE:-/tmp/engineering-skills-jscpd-cache}"
RUN_TS_JSCPD="$SKILL_ROOT"/scripts/run_jscpd.py
COLLAPSE_TS="$SKILL_ROOT"/scripts/collapse_typescript.py
RANK="$SKILL_ROOT"/scripts/rank.py
REPORT="$SKILL_ROOT"/scripts/report.py
mkdir -p "$REPORT_DIR/jscpd"

"$PYTHON" "$RUN_TS_JSCPD" \
  --target "$TARGET" --output "$REPORT_DIR/jscpd" --npm-cache "$NPM_CACHE"
"$PYTHON" "$COLLAPSE_TS" \
  --jscpd-report "$REPORT_DIR/jscpd/jscpd-report.json" \
  --target "$TARGET" --project-root "$PWD" \
  --output "$REPORT_DIR/collapsed.json"
"$PYTHON" "$RANK" \
  --input "$REPORT_DIR/collapsed.json" --output "$REPORT_DIR/ranked.json"
"$PYTHON" "$REPORT" \
  --input "$REPORT_DIR/ranked.json" \
  --output-md "$REPORT_DIR/triage.md" \
  --output-json "$REPORT_DIR/findings.json" \
  --scan-id "$SCAN_ID"
ln -sfn "$SCAN_ID" reports/duplication/latest

Required artifacts are jscpd/jscpd-report.json, collapsed.json, ranked.json, triage.md, and findings.json. The final Markdown repeats “Do not consolidate automatically.” A nonempty cluster is an investigation lead only. unmapped_symbol, span_crosses_symbol_boundary, overload, and excluded-path omissions are deliberate false-negative boundaries.

JavaScript lexical-evidence branch

JavaScript v1 accepts .js, .jsx, .mjs, and .cjs only through an explicit project-local jscpd binary. It never runs npm or npx and never installs a tool. The runner emits run.json with tool-missing, syntax-error, tool-failed, or partial when an established final clone report cannot be produced; none of those outcomes is clean.

The collapse pass retains a reported pair only when both spans fit a named function or block-bodied arrow. It excludes generated, minified, test, vendor, dependency, report, staging, and symlink paths and maps source lines from the original host files. The final triage.md says “Do not consolidate automatically”; it is lexical evidence, not a behavior, caller, or semantic equivalence conclusion.

PYTHON="${PYTHON:-python3}"
RUN_JS_JSCPD="$SKILL_ROOT"/scripts/run_jscpd_javascript.py
COLLAPSE_JS="$SKILL_ROOT"/scripts/collapse_javascript.py
JSCPD_BIN="$PWD/node_modules/.bin/jscpd"

"$PYTHON" "$RUN_JS_JSCPD" --target "$TARGET" --project-root "$PWD" --output "$REPORT_DIR/jscpd" \
  --jscpd-bin "$JSCPD_BIN" || exit $?
"$PYTHON" "$COLLAPSE_JS" --jscpd-report "$REPORT_DIR/jscpd/jscpd-report.json" \
  --target "$TARGET" --project-root "$PWD" --output "$REPORT_DIR/collapsed.json" || exit $?
"$PYTHON" "$RANK" --input "$REPORT_DIR/collapsed.json" --output "$REPORT_DIR/ranked.json" || exit $?
"$PYTHON" "$REPORT" --input "$REPORT_DIR/ranked.json" \
  --output-md "$REPORT_DIR/triage.md" --output-json "$REPORT_DIR/findings.json" \
  --scan-id "$SCAN_ID"

Go exact-function evidence branch

Go v1 uses Go 1.22+ from PATH and one batched go run of the bundled go/parser standard-library helper. It fingerprints go/format-normalized bodies of named functions and receiver methods with at least five source lines, then retains only fingerprints occurring at two or more symbols. This is exact structural evidence, not semantic equivalence, caller proof, or a safe-reuse recommendation.

The source inventory is project-root-relative and excludes _test.go, generated, test/testdata/fixture, vendor, dependency, report, and build-output trees even when one is targeted directly or through a symlink. Generated files are excluded before build classification. Explicit build tags and implicit GOOS/GOARCH filename constraints make an otherwise useful result partial; malformed source, missing/old Go, or invalid helper evidence is failed or unsupported, never clean.

PYTHON="${PYTHON:-python3}"
SKILL_ROOT="${SKILL_ROOT:-.claude/skills/find-duplication}"
SCAN_ID="scan-$(date -u +%Y%m%d-%H%M%S)"
REPORT_DIR="reports/duplication/${SCAN_ID}"
mkdir -p "$REPORT_DIR"

"$PYTHON" "${SKILL_ROOT}/scripts/run_go.py" \
  --target src --project-root "$PWD" --output "$REPORT_DIR/collapsed.json"
"$PYTHON" "${SKILL_ROOT}/scripts/rank.py" \
  --input "$REPORT_DIR/collapsed.json" --output "$REPORT_DIR/ranked.json"
"$PYTHON" "${SKILL_ROOT}/scripts/report.py" \
  --input "$REPORT_DIR/ranked.json" \
  --output-md "$REPORT_DIR/triage.md" \
  --output-json "$REPORT_DIR/findings.json" --scan-id "$SCAN_ID"

The final Markdown says “Do not consolidate automatically.” Review both bodies and their callers before proposing a refactor.

Java exact-method evidence branch

Java v1 uses java and javac from JDK 17+ and one batched source-launcher invocation of the family-local JDK compiler-tree helper. It fingerprints the normalized bodies of direct methods and constructors on named top-level types when the complete declaration spans at least five lines. Exact fingerprints at two or more symbols become review leads; this is not semantic equivalence, caller proof, type resolution, inheritance analysis, or a safe-reuse claim.

Tests, generated source, fixtures, vendor/dependency, report, and build-output trees are excluded. The parser does not run annotation processors or infer Lombok/framework-generated members. Malformed source, missing/old JDK, or invalid helper evidence is failed/unsupported, never clean.

PYTHON="${PYTHON:-python3}"
SKILL_ROOT="${SKILL_ROOT:-.claude/skills/find-duplication}"
SCAN_ID="scan-$(date -u +%Y%m%d-%H%M%S)"
REPORT_DIR="reports/duplication/${SCAN_ID}"
mkdir -p "$REPORT_DIR"

"$PYTHON" "${SKILL_ROOT}/scripts/run_java.py" \
  --target src --project-root "$PWD" --output "$REPORT_DIR/collapsed.json"
"$PYTHON" "${SKILL_ROOT}/scripts/rank.py" \
  --input "$REPORT_DIR/collapsed.json" --output "$REPORT_DIR/ranked.json"
"$PYTHON" "${SKILL_ROOT}/scripts/report.py" \
  --input "$REPORT_DIR/ranked.json" --output-md "$REPORT_DIR/triage.md" \
  --output-json "$REPORT_DIR/findings.json" --scan-id "$SCAN_ID"

The final Markdown retains “Do not consolidate automatically.” Review the matched bodies and their callers before proposing any refactor.

Mixed targets

For a mixed repository, use one outer scan ID and separate branches:

reports/duplication/<scan-id>/python/...
reports/duplication/<scan-id>/typescript/...
reports/duplication/<scan-id>/javascript/...
reports/duplication/<scan-id>/java/...

Run Python with its AST + scout stages and JavaScript/TypeScript with their conservative evidence paths. Produce separate final reports and summarize them under their own claims. Do not concatenate their ranked JSON.

Failure handling

Non-goals

• Editing source or executing a refactor.
• Treating dormant code as a primary duplication finding.
• Turning TypeScript lexical evidence into semantic equivalence.
• Turning JavaScript lexical evidence into semantic equivalence.
• Turning exact Java body fingerprints into semantic equivalence or safe reuse.
• Creating a shared parser, detector service, or cross-family runtime.