Test Fleet

Test Fleet

Autonomous test-coverage backlog sweep — enumerate under-covered areas and uncovered critical paths, confirm a ranked target batch with the human in one up-front round, fan out worktree-isolated subagents that each author tests via the real authoring flow (tdd then test-craft), independently verify every result by added behavior-asserting tests plus a coverage delta plus all-OS CI, and hand back a batch of green test PRs for one bulk review. The fleet never auto-merges and never trusts a subagent's self-report.

Closing a test-coverage backlog by hand is the same attention slog the other -fleet members attack, in the quality domain. Every under-covered area has to be found (which modules and critical paths lack behavior-covering tests?), then have tests authored through the real flow, then run green on every platform, then shipped as a reviewable PR — one area at a time, with a human at each step. For a codebase with dozens of gaps the human's attention is the bottleneck, not the machinery. test-fleet is a quality-queue member of the -fleet family, working the test-coverage backlog alongside the core conveyor (intake → decide → build → land). It is the structural twin of roadmap-fleet: both fan out over a work-queue with a concurrency governor and independent verification, then stop at reviewable PRs they never merge. The difference is the queue (coverage gaps, not backlog items), the per-item pipeline (test authoring, not feature build), and the shape of the verification artifact (authored tests plus a coverage delta, not a plan directory). The shared, stage-agnostic scaffolding it builds on is documented in the -fleet family spine reference (docs/reference/fleet-family.md).

When to Use

• A test-coverage backlog of under-covered areas needs autonomous test authoring plus one bulk PR review instead of per-area babysitting
• Clearing accumulated coverage debt where authoring tests for each area by hand does not scale
• Turning a coverage audit's list of gaps and uncovered critical paths into a batch of green, reviewable test PRs in a single session
• When the gaps are largely independent — each area's tests land on their own and do not depend on another area's tests first
• NOT for authoring a single test — invoke harness-tdd (or a single-test author) directly; a fleet's overhead only pays off across a backlog of gaps
• NOT for fixing the code under test — test-fleet characterizes existing behavior; if a gap exposes a bug it parks that target and reports it, it does not fix the code (that is the build pipeline)
• NOT for chasing a coverage percentage with assertion-free tests — coverage theater is rejected; a "covered" line must be exercised by a behavior-asserting test
• NOT for converging one module's tests to done through repeated rounds — that is a pipeline (it loops on one target), not a fleet (which fans out across many independent gaps into many PRs)

Flags

Process

Iron Law

A test PR is "merge-ready" only after independent verification that it adds behavior-asserting test files, improves the target's coverage, and is CI-green across all OS. The fleet never auto-merges, and never accepts a subagent's self-report as proof its authoring flow ran.

A subagent that reports "done — tests written, coverage up, CI green" has told you what it believes, not what is true. The only evidence that the real authoring flow ran is the artifact it necessarily leaves behind — new or changed test files on the branch and a measurable coverage delta for the target — plus the CI signal on the pushed branch. A target with no added test did not run the flow; a coverage number that rose without behavior-asserting tests is coverage theater. Either way the target is rejected or retried, regardless of how confident the report reads. And shipping the batch is the human's call: the fleet stops at a set of verified, reviewable test PRs.

Phase 1: SELECT --> Phase 2: CONFIRM --> Phase 3: DISPATCH
                                                    |
                                                    v
                     Phase 5: REPORT <-- Phase 4: VERIFY

The five-phase spine, the concurrency governor, the artifact + all-OS-CI verification discipline, the worktree fan-out with its push-path caveat, and the never-silent-merge invariant are the family-shared scaffolding — stated once in docs/reference/fleet-family.md. This skill states only what is specific to the test-coverage stage: the coverage-gap queue, the tdd-then-test-craft per-item authoring flow, and the authoring-shaped verification artifact.

Phase 1: SELECT — Enumerate, Cross-Check, Score

1. Enumerate coverage gaps from both sources. Compose harness-test-advisor's project-wide coverage audit ("what's untested?") for under-covered modules/files, together with the graph's uncovered critical paths (a critical path with no behavior-asserting test). Each gap becomes a candidate target — a module/file or an uncovered critical path. A missing coverage tool or coverage data degrades to whichever source is available; record which source was unavailable rather than aborting.

2. Define "covered" honestly. A target is covered only when it is exercised by at least one test that asserts behavior — not merely imported, and not executed for line credit without assertions. This definition is what SELECT ranks against and what VERIFY re-checks; it is the guard against coverage theater.

3. Cross-check each target against in-flight and merged coverage. For every target, check whether a test PR already in flight covers it (in-progress-elsewhere — drop it from the batch) or whether recently-merged tests already cover it (already-covered — annotate, do not re-sweep). Only novel gaps proceed.

4. Score and order via roadmap-pilot impact scoring. Do not rank ad-hoc. Reuse harness-roadmap-pilot-style impact scoring with a criticality × coverage-deficit weighting so selection is principled and reproducible; uncovered critical paths rank highest, a large deficit on a low-criticality util ranks low. Order the batch highest-impact first.

5. Detect decision forks up front. For each target, scan for genuine ambiguity a test author would otherwise have to guess (e.g. "characterize the current behavior as-is, or is this target expected to change soon?", "is this observable output part of the contract or an implementation detail?"). Surface these known forks in CONFIRM; do not answer them here.

6. Build the CoverageTarget record for each survivor:

   CoverageTarget {
     id,                // module/file path or critical-path id
     title,
     criticality,       // graph criticality signal
     coverageDeficit,   // gap size from the coverage audit
     score,             // roadmap-pilot impact score (criticality x deficit)
     crossCheck,        // "novel" | "already-covered" | "in-progress-elsewhere"
     groupWith,         // PR-grouping hint for small cohesive targets (may be empty)
     forks,             // detected known decision forks (may be empty)
   }
   

Phase 2: CONFIRM — The Single Up-Front Human Gate [checkpoint:human-verify]

1. Present the ranked target batch in one round. This is the only guaranteed human touchpoint before PR review — everything downstream runs autonomously. Present, together, in a single surface:

   • The ranked targets (highest-impact first) with scores and coverage deficits.
   • Already-covered / in-flight targets flagged for drop or annotation — the fleet never re-sweeps them.
   • Every detected known decision fork as a multiple-choice question with a recommended default.
   • The proposed PR grouping — which small, cohesive targets fold into one review-sized PR and which get their own.
   • The proposed concurrency (default 2, capped at ~3).

2. The human approves or trims once, and answers the forks. Batch approval, fork answering, drop confirmation, and grouping approval all happen in this same gate — front-loading the genuinely-ambiguous items is what keeps wrong-guess rework low. Answered forks are recorded and fed into each target's DISPATCH brief.

3. From here it is autonomous. After this gate the fleet does not pause per-target. The only thing that re-surfaces to the human before REPORT is an unforeseen fork that parks a single target (see DISPATCH) — and even that does not block the batch. Under --dry-run the skill stops at the end of this phase.

Phase 3: DISPATCH — Worktree Authoring Fan-Out With a Concurrency Governor

1. One worktree-isolated subagent per confirmed target. Each subagent is briefed to run the real per-target authoring flow: harness-tdd to author behavior-covering tests for its one target, then test-craft to critique and raise their quality (contract-vs-implementation, assertion strength, no brittle coupling). Where the project ships a dedicated test-authoring plugin (e.g. canary), the subagent may compose it — but the authoring flow is the hard dependency so the skill runs standalone in any adopter project. Feed the target's answered forks from CONFIRM into the brief so the author never re-asks a settled question.

2. Author behavior, never coverage theater, never edit the code under test. The subagent writes tests that assert behavior — not import-only or assertion-free tests that raise the number without proving anything. It characterizes existing behavior; it does not modify the code under test to make a test pass. If a gap reveals a probable bug, that is a park (below), not a code edit.

3. Cap concurrency at the governor (default 2, max ~3). This is the machine-storm limit shared across the family: beyond roughly three concurrent authoring agents the compound load produces flaky failures indistinguishable from real ones. Never exceed the confirmed concurrency to "go faster" — a stormed batch is slower once you account for re-runs.

4. Park unforeseen forks; never guess mid-flight. A subagent runs autonomously on recommended-option defaults for anything routine. But if a target hits a genuinely unforeseen decision fork — the gap exposes a probable bug, or the "correct" behavior is genuinely ambiguous and characterizing it as-is would bake in a wrong contract — that target parks and reports the fork instead of guessing. Parking is per-target: the other targets continue uninterrupted. The parked fork appears in REPORT for the human.

5. Record an "assumptions made" note per target. Each subagent records the recommended-option defaults it took (e.g. "characterized current behavior as-is") so the eventual PR carries an assumptions note — batch review is only trustworthy when the reviewer can see what was assumed.

6. Push-path caveat. A worktree created under a .claude/-nested path breaks the local pre-push check-docs gate (it self-excludes and scans zero files). Subagents push via the GitHub API or from a non-.claude throwaway worktree. Never --no-verify — bypassing the gate defeats the verification the fleet depends on.

Worker handoff — return the canonical FleetHandoffRecord. When a worker finishes its target it hands the orchestrator exactly one FleetHandoffRecord (from @harness-engineering/types) — the ONE bounded envelope every -fleet member emits, so fleet-command parses any fleet's worker output uniformly instead of special-casing an ad hoc per-worker report shape. The record carries status (done | parked | blocked | failed), fleet, item, a one-line summary, an evidence[] of verifiable pointers (branch, PR, artifact path, CI check — exactly the references VERIFY re-checks), next_steps[], and, for any non-done status, a blocker. The orchestrator validates it with validateFleetHandoffRecord; a malformed or unknown-keyed record is rejected, never silently misread. See the canonical handoff record in docs/reference/fleet-family.md.

Phase 4: VERIFY — Independent Confirmation, Never Self-Report

1. Never accept a subagent's self-report as verification. "Tests written, coverage up, CI green" is a claim to be checked, not a result. For each returned branch, the orchestrator independently confirms the evidence itself.

2. Require added behavior-asserting test files. Confirm the branch actually adds or changes test files for the target, and that those tests assert behavior rather than merely importing or executing lines. A target with no added test did not run the authoring flow — regardless of what the subagent reported. Reject it (or retry once); it is never marked merge-ready.

3. Require an improved coverage delta. Re-audit the target's coverage (via harness-test-advisor) and confirm it improved. A branch whose coverage did not move — or moved only through assertion-free tests — is coverage theater, not coverage. Reject it.

4. Require all-OS CI green. Confirm the pushed branch's CI is green on all three operating systems plus the enforce and harness checks, and that the full test suite (not just the new tests) still passes — the new tests must pass everywhere and break nothing existing. Green on one OS is not green. A subset-red branch is reported failed, and the batch continues.

5. Classify each returned target as verified (added behavior-asserting tests + improved coverage delta + all-OS CI green), rejected (no added test, coverage theater, or definitively red), or retry (transient, retried at most once). No target reaches REPORT as merge-ready without passing all three checks.

Phase 5: REPORT — Batch Summary, Coverage Deltas, Never Merge

1. Emit a one-row-per-target batch summary for bulk human review:

   Target	Verdict	PR	Coverage delta	Assumptions made	Parked forks

   Every verified target's row carries its PR link, the coverage delta it added, the assumptions-made note from DISPATCH, and any parked unforeseen fork. Rejected/failed targets are listed with the reason (no added test, coverage theater, subset-red CI). Grouped targets share one PR row.

2. Annotate already-covered targets accurately. For each target flagged already-covered in SELECT and confirmed in CONFIRM, record that it was skipped as already-covered — never re-swept, never a duplicate test PR.

3. Never merge. The fleet delivers verified, reviewable test PRs; the human (optionally via pr-fleet) lands the batch. Auto-merging a test PR is out of scope by design.

4. Degrade gracefully. A missing coverage tool, a single target's failed authoring, or a gap that exposed a bug results in that target being reported/parked while the rest of the batch proceeds. One bad target never sinks the batch.

Harness Integration

• harness skill run test-fleet — Run the full five-phase batch pipeline.
• harness-test-advisor — Composed in SELECT to enumerate coverage gaps and uncovered critical paths, and in VERIFY to re-audit each target's coverage delta; the fleet does not reimplement coverage analysis.
• harness-roadmap-pilot — Its impact scoring is reused in SELECT to order targets by criticality × coverage-deficit.
• harness-tdd / test-craft — The real per-target authoring flow each subagent runs in DISPATCH (author behavior-covering tests, then critique their quality); the fleet composes them and never reimplements authoring.
• gh — Enumerate in-flight/merged test PRs for cross-check (SELECT), read gh pr checks (VERIFY), and open the batch of reviewable test PRs (REPORT).
• docs/reference/fleet-family.md — The shared -fleet spine this skill builds on (five-phase skeleton, concurrency governor, artifact + all-OS-CI verification discipline, worktree fan-out, never-silent-merge).
• harness skill validate test-fleet — The authoring-time gate for this skill's own structure and schema.

Success Criteria

• Given a confirmed batch of N coverage targets, the fleet produces up to N test PRs (some grouped), each with added behavior-asserting tests, an improved coverage delta, and green CI across all three OS plus enforce and harness.
• There is exactly one up-front human decision round; no per-target interactive pauses except a genuinely-new fork parked to its own target.
• Every emitted PR carries an "assumptions made" note and its coverage delta.
• Coverage-theater tests (assertion-free, import-only) are rejected — a "covered" target must gain a behavior-asserting test.
• Already-covered / in-flight targets are dropped or annotated, not re-swept.
• The skill never auto-merges a test PR.
• It degrades gracefully: a missing coverage tool, a single target's failed authoring, or a gap that exposes a bug is reported/parked while the batch continues.
• Concurrency never exceeds the confirmed governor (default 2, max ~3).
• No target is marked merge-ready on a subagent self-report — every verdict is backed by independently-checked test-file + coverage-delta + CI evidence.

Gates

• No "merge-ready" without added behavior-asserting test files. A target whose branch adds no test — or only assertion-free tests — did not run the real authoring flow. It is rejected or retried, never reported as merge-ready, no matter what the subagent claimed.
• No "merge-ready" without an improved coverage delta. A branch whose coverage did not move is coverage theater; a rising number from assertion-free tests is not coverage. Report it rejected; do not ship it.
• No "merge-ready" without all-OS CI green. Green on a subset of operating systems (or with enforce/harness red, or the existing suite broken) is not merge-ready. Report it failed; do not ship it.
• Never edit the code under test. test-fleet characterizes behavior; changing the code to make a test pass — or to "fix" a bug the gap exposed — is out of scope. Park the target and report it.
• Never auto-merge. The fleet stops at reviewable test PRs. Merging a test PR from inside the fleet = gate violation; the human lands the batch.
• A self-report is never verification. Accepting "tests written, coverage up, CI green" without independently checking the test files, the coverage delta, and CI = gate violation. Re-verify independently.
• Never exceed the concurrency governor. More than ~3 concurrent authoring agents is the machine-storm zone; do not raise the cap to "go faster."
• Never --no-verify. No subagent bypasses the pre-push gates; a .claude/-nested worktree pushes via the GitHub API or a non-nested worktree instead.

Escalation

• Missing coverage tool / no coverage data: proceed with whichever source is available (e.g. critical-paths only); record the missing source in REPORT rather than aborting the batch. If neither coverage nor critical-path data is available, stop and report — there are no gaps to sweep.
• A subagent returns a branch with no added test: do not accept its self-report. Reject or retry once; if it still adds no behavior-asserting test, report the target as "did not run the authoring flow" and move on — the batch continues.
• A target's coverage did not improve (coverage theater): report it rejected with the reason; never mark it merge-ready on a risen line-count alone.
• A gap exposes a probable bug: the target parks — test-fleet does not fix the code. Surface the suspected bug (with the target's context) in REPORT for the human to route to the build pipeline; the parked target is the only one affected.
• CI red on a subset of OS (or the existing suite broken): report the target failed with the failing OS/check named; never mark it merge-ready. Do not average a mixed CI result into "mostly green".
• The batch appears coupled (one target's tests depend on another's merge): stop fanning out those targets; the coupling means they are a pipeline, not a fleet. Escalate to the human to sequence them.

Rationalizations to Reject

Red Flags

Examples

Example: A six-gap coverage backlog batch

$ harness skill run test-fleet --concurrency 2

Phase 1: SELECT
  Enumerated via test-advisor coverage audit + graph critical paths:
    6 under-covered targets + 2 uncovered critical paths.
  Cross-check vs in-flight/merged test PRs:
    - "token-store.ts"      -> already-covered (merged test PR) -> annotate, drop
    - "webhook-retry.ts"    -> in-progress-elsewhere (open test PR) -> dropped
  Scored 6 survivors via roadmap-pilot (criticality x deficit); critical paths ranked top.
  Detected forks: 1 — "auth-guard: characterize current 401 behavior as-is, or is it changing?"

Phase 2: CONFIRM  [checkpoint:human-verify]
  Ranked batch (6) presented. 2 targets flagged already-covered/in-flight.
  Human answers fork: auth-guard -> characterize as-is.
  Human trims 1 low-value util target -> batch = 5. Grouping: 2 small parsers -> 1 PR.
  Concurrency confirmed: 2.

Phase 3: DISPATCH (governor = 2)
  4 worktree-isolated subagents (2 grouped), 2 at a time, each running real
  tdd -> test-craft for its target; answered forks fed into briefs.
  Target "rate-limiter.ts" hits an UNFORESEEN fork (its behavior looks like a bug)
    -> parks and reports; the others continue.

Phase 4: VERIFY (independent — no self-report)
  auth-guard:  added behavior-asserting tests, coverage +14%, CI green all 3 OS + enforce + harness -> verified
  parsers (grouped): added tests, coverage +9%, CI green all 3 OS -> verified
  cache.ts:    tests added but coverage delta ~0 (assertion-free, import-only) -> REJECTED (coverage theater)
  rate-limiter.ts: parked in DISPATCH -> not verified (fork awaits human)

Phase 5: REPORT
  | Target        | Verdict  | PR   | Coverage delta | Assumptions made          | Parked forks           |
  | ------------- | -------- | ---- | -------------- | ------------------------- | ---------------------- |
  | auth-guard    | verified | link | +14%           | characterized 401 as-is   | —                      |
  | parsers (x2)  | verified | link | +9%            | —                         | —                      |
  | cache.ts      | rejected | —    | ~0%            | —                         | — (coverage theater)   |
  | rate-limiter  | parked   | —    | —              | —                         | behavior looks like bug|
  Annotated 2 already-covered targets. Never merged. 2 test PRs handed over for review.

Example: Rejecting a coverage-theater target

A subagent returns a branch and reports "done — added tests, coverage is up, CI green." VERIFY re-audits the target: the branch does add test files, but they only import the module and assert nothing, so the coverage delta is ~0 in behavior terms. Per the Iron Law the target is rejected (retried once, still assertion-free → reported as "coverage theater"), never marked merge-ready. The batch's other verified targets proceed to REPORT unaffected.

Test Scenarios

Scenario 1: Gate — a self-report accepted as verification

VERIFY receives a subagent claiming "tests written, coverage up, CI green" but the branch adds no behavior-asserting test for the target. Expected: the "no merge-ready without added behavior-asserting test files" Gate halts marking it merge-ready; the target is rejected/retried, not reported as a PR. Accepting the self-report is the failure this scenario guards against.

Scenario 2: Rationalization — coverage theater to bump the number

A subagent writes import-only, assertion-free tests so the coverage percentage rises. Expected: rejected by the "no merge-ready without an improved coverage delta" gate — the re-audit shows no behavior was covered. The correct action is to author tests that assert behavior via the real tdd/test-craft flow.

Scenario 3: Park-unforeseen — a gap that exposes a bug

An authoring subagent finds that the target's current behavior looks like a bug, so characterizing it as-is would bake in a wrong contract. Expected: the target parks and reports the suspected bug rather than editing the code to make a test pass or guessing the "right" behavior; the parked fork appears in REPORT for the human to route to the build pipeline; the other in-flight targets continue uninterrupted. Editing the code under test is the failure this scenario guards against.

Scenario 4: Self-report — accepting "CI green" without checking all OS

A subagent reports "CI green." Expected: the "a self-report is never verification" gate requires the orchestrator to independently confirm all-OS CI (plus the existing suite still passing) via gh before marking merge-ready. Marking it ready on the report alone — or on one OS being green — is the failure this scenario guards against.