# Optimizing Submodules

> Minimize submodule sparse-checkouts to only the trees the repo actually consumes.

- Skill: `socketdev/optimizing-submodules` (Agent Skill)
- Install (CLI): `npx skillmds@latest add socketdev/optimizing-submodules`
- Raw SKILL.md: https://api.skillmd.com/api/skills/socketdev/optimizing-submodules/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: socketdev (https://skillmd.com/u/socketdev)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/socketdev/optimizing-submodules

---


# optimizing-submodules

A vendored `upstream/<name>` submodule is rarely consumed in full. It is a parser reference, a test corpus, one subdir of a build, or a pin-only crate whose code actually comes from a registry. Without a `sparse-checkout`, the whole upstream tree lands in every clone (test262 alone is ~270 MB, typescript-go's `testdata/baselines` is 283 MB). This skill restricts each submodule to what the repo reads.

The `submodules-are-sparse-or-annotated` gate (`scripts/fleet/check/`) fails `check --all` for any submodule that is neither sparse nor annotated `# full-checkout: <reason>`. This skill is how you satisfy it.

## The discipline: determine → apply → verify

**The determination is judgment (AI-assisted); the application is law (scripted).** Propose the pattern by analysis, prove it by building.

### 1. Determine (analyze what consumes the submodule)

First gather the evidence — one deterministic pass that `rg`s every submodule, applies the outside-only filter, and buckets the surviving hits by file type:

```bash
node scripts/fleet/optimizing-submodules/collect-submodule-consumers.mts --pretty
```

It emits, per submodule, the OUTSIDE-only consumer hits bucketed `rust` / `cpp` / `go` / `jsts` / `testCorpus` / `build` / `other`, the current sparse/verify state, the on-disk tree size (the why-bother signal), and an `internalHitCount` — the self-references it excluded. Plain (no `--pretty`) emits the same as a JSON envelope. It renders **no verdict** — that is your call from the buckets. Read each bucket and interpret it:

- Rust (`Cargo.toml` / `build.rs`): a **path/git** dep is consumption; a `version = "=x"` crates.io pin is NOT — the code comes from the registry, the submodule is reference-only.
- C/C++ (`CMakeLists.txt` / `binding.gyp`): which subdir does `add_subdirectory` / a `*_DIR` var point at?
- Go (`go.mod`): a registry module is not the submodule.
- JS/TS (imports / `package.json` / vitest): a `workspace:*` fork supersedes the submodule.
- Test corpora: in the conformance runner under `test/`, find the **exact fixture subdir** it walks.
- Build/bench scripts.

**Trap — internal self-references (now handled by the collector).** A vendored crate's own files reference each other (e.g. blake3's `b3sum/Cargo.toml` has `path = ".."`). Those are the submodule consuming itself, NOT your repo consuming it. The collector already drops every hit inside `upstream/<name>/` (the `isInsideSubmodule` filter) and reports the count as `internalHitCount`, so the over-check that filter prevents is code, not a hand-discipline.

Outcomes per submodule:
- **Subtree-consumed** → the minimal pattern (e.g. `c`, `src`, `tests files-toml-1.0.0`, `files`).
- **Reference-only** (pin tracking, crates.io/npm dep, lockstep metadata, a doc cites it) → a minimal `README.md` (or the specific cited files) so the dir isn't empty but pulls ~nothing.
- **Genuinely whole-tree** — a crate built from its entire source with no separable subtree → no sparse; annotate the block `# full-checkout: <reason>`.

### 2. Apply (scripted)

Write the pattern into `.gitmodules`, which is what `git-partial-submodule.mts clone` honors:

```bash
git config -f .gitmodules submodule."<name>".sparse-checkout "<space-separated patterns>"
```

For a populated submodule, also re-narrow the working tree and persist:

```bash
git -C <path> sparse-checkout set <patterns>
node scripts/fleet/git-partial-submodule.mts save-sparse <path>   # writes the field from the live state
```

`add --sparse` (clone sparse) and `restore-sparse` (re-apply the recorded field) are the other primitives.

### 3. Verify (prove it by building — this step is law, not habit)

A too-narrow pattern breaks the build only at use, so static analysis can pass it through. The verify is enforced by code, not left to discretion. Declare the consumer in `.gitmodules` next to the sparse pattern:

```
verify = pnpm --filter @x/parser test     # the command that builds against the subtree
verify = none                              # reference-only — nothing builds against it
```

Then prove it: `verify-submodule-sparse.mts --run <name|path>` sparse-clones the submodule per its recorded pattern and runs the declared `verify =` command. Green → the pattern is build-sufficient. Fail → it's too narrow — a needed path isn't checked out; widen and re-run.

```bash
node scripts/fleet/verify-submodule-sparse.mts --run <name|path>   # prove one
node scripts/fleet/verify-submodule-sparse.mts --run-all           # CI / on-cadence (heavy: clone + build each)
node scripts/fleet/verify-submodule-sparse.mts --check             # gate: every sparse block declares a verify =
```

The `--check` gate (in `check --all`) fails any sparse block with no `verify =` — a pattern with no declared consumer is unproven, so it can't land. That is what makes the verify law: you cannot add a sparse pattern without naming how it is build-proven.

## Removing a submodule

```bash
git submodule deinit -f <path>
git rm <path>
git config -f .gitmodules --remove-section submodule."<name>"   # if any residue remains
```

Commit `.gitmodules` + the gitlink removal together.

## Gate

`node scripts/fleet/check/submodules-are-sparse-or-annotated.mts` — green when every block is sparse or `# full-checkout:`-annotated. Run it after the sweep; it is in `check --all`.