Test Auto Fix

test-auto-fix

Autonomously diagnose and fix failing tests in any project, with hard guardrails so the loop never devolves into "make the red go away".

This SKILL.md is the orchestration index. Load the matching rule file when you need detail — do not preload them.

Always read rules/anti-patterns.md first. The hard refusals apply to every phase.

Input

The user provides one of:

• Nothing — auto-run every surface defined in the project surface file.
• A surface name (vitest, unit, integration, etc.) — run only that surface.
• A file path — narrow detection and re-run to that file.
• --surface <path> — override the surface file lookup entirely.
• --plan-only — diagnose and produce the plan artifact; do not write code.
• --max-iterations N — outer-loop cap (default 5).

The argument is: $ARGUMENTS.

Phase 0 — Resolve the surface

Determine the surface file for the current project. Phase 0 is a hard two-branch decision; pick exactly one path and follow it through.

Step 1 — Pick the resolution branch

• If --surface <path> was passed → branch A.
• Otherwise → branch B.

Branch A — --surface <path> override

Use the file at <path> directly as the surface. Do NOT compute the project key, do NOT scan surfaces/, do NOT run bootstrap.

Still validate per rules/surface-validation.md, but tolerate the project-key mismatch warning (it's expected when overriding).

When validation passes, jump straight to Phase 1.

Branch B — Auto-resolve from project key

1. Compute the project key per rules/project-keying.md.

2. Look for surfaces/<project-key>.md next to this skill file. The skill's own directory is resolved by following the symlink chain of the loaded SKILL.md:

   readlink -f "$(dirname "$0")"
   

   If that fails (model-invocation context), default to ~/.agents/skills/test-auto-fix/surfaces/.

3. If no surface file is found → run bootstrap per rules/bootstrap.md. Bootstrap detects the stack, proposes a surface diff, waits for user approval, then writes the file. Do NOT proceed to Phase 1 until the surface exists.

4. If a surface file exists → validate it per rules/surface-validation.md. If validation fails, propose an update diff and ask once. If the user declines the update, escalate — do not run with an invalid surface.

Phase 1 — Detect failures and build the plan

1. Run the detect command from the surface file:

   <detect-command>
   

   If --path <file> was supplied, append it to narrow the run. If the cache-bust-flag is set in the surface and a prior run was cached-green, append the flag to force a real run.

2. Parse failures using the failure-parser regex from the surface file. Group by file, then by failure type.

3. If every surface is green: stop immediately. Tell the user. Do not invent work.

4. Write the fix plan to .agent/{branch}/test-auto-fix-plan.md using templates/plan-artifact.md. The plan is read-only documentation of intent. Order: high-confidence test-bug fixes first, prod-bug suspects last.

Phase 2 — Classify each failure (verdict required)

For every failure, apply the rubric in rules/verdicts.md.

Getting this wrong means later phases can corrupt production code to satisfy a broken test, or "fix" a real regression by patching the test.

Emit exactly one verdict per failure:

• test-bug — production code correct; test drifted. Continue to Phase 3.
• prod-bug — production code regressed; test caught it. Escalate. Do not touch the test.
• unsure — confidence < 80%. Escalate. Surface the evidence and stop.

Phase 3 — Draft the minimal fix

For each test-bug failure, draft the smallest edit that addresses the diagnosed root cause.

Hard refusals — full list and detection patterns in rules/anti-patterns.md:

• Deleting any test, describe, it, or story export.
• Adding .skip, .only, xit, xdescribe, fit, fdescribe, it.todo, test.todo.
• Replacing strict matchers with loose ones.
• Deleting expect(...) / assert*() calls.
• Mocking the System Under Test (the module being tested) instead of its deps.
• Wrapping the failing assertion in try/catch to swallow it.

Phase 3.5 — Confidence gate (before any edit)

Invoke the confidence skill in analysis mode for each proposed fix:

Skill("confidence", "analysis proposed fix: <one-line summary>; verdict: <test-bug>; surface: <surface-name>; risk: <test-only|prod-touch>")

Record the score in the plan artifact.

Full decision matrix: rules/confidence-gate.md.

At a glance:

The gate is non-negotiable. Auto mode does not override it.

Phase 4 — Apply + verify (per fix)

1. Apply the edit with Edit — never Bash sed/awk.

2. Re-run only the single failing test using the single-test-command from the surface file, substituting {file} and {name}:

   <single-test-command>
   

3. If still failing: do not patch over. Revert with:

   git restore <file>
   

   Re-classify at Phase 2.

4. If green, and test-provenance-guard is installed: invoke Skill("test-provenance-guard") on the file. If it reports tests-by-construction, revert and escalate.

5. Only after both checks pass, mark the failure resolved in the plan.

Phase 5 — test-provenance-guard (optional)

If Skill("test-provenance-guard") is not installed, log: companion: test-provenance-guard — not available, continuing

Phase 6 — Outer loop

After a batch of fixes:

1. Re-run the full surface using the detect command:

   <detect-command>
   

   Regressions hide in untouched files — always re-run the full surface.

2. Compare against the previous failure set per rules/regression-detection.md:

   • Same failures → back to Phase 2 (re-classify; do not repeat same fix).
   • Strict subset → continue with remaining failures.
   • New failure → revert last fix, re-plan or escalate.

3. Cap at --max-iterations (default 5).

4. On any exit, write a summary section to the plan file.

Phase 7 — Report

Always end with the structured exit summary from templates/exit-summary.md.

test-auto-fix run
  Outcome: <green | escalated | regression-reverted | max-iterations>
  Resolved: <N> failures
  Escalated: <N> failures (<verdicts>)
  Iterations: <N>/<max>
  Surface: <surface-file-path>
  Plan: .agent/{branch}/test-auto-fix-plan.md
  Escalation reason: <…>           # if not green

Self-Improvement

/test-auto-fix gets better across runs through a two-tier lessons loop (fast episodic tier + gated promotion), like autonomous-workflow and fix-bug. It reads test-auto-fix-lessons at Phase 2 (biasing the verdict and the Phase 3 fix sub-class) and writes at Phase 6 (regression / same-failure signal) and Phase 7 (outcome retrospective), keyed by stack : failure-pattern : verdict-sub-class. This complements the surface file (which is config — how to run tests here — not learned judgment) rather than duplicating it. Lessons are advisory — they never lower the confidence gate, never turn a prod-bug/unsure escalation into a silent test edit, and never override a refusal in rules/anti-patterns.md.

Most value is within a project — the repo::{owner}/{repo} scope (catching a recurring verdict misclassification for a failure shape); cross-project leverage (the global scope) is weaker because the feedback is binary and local. A recurring lesson (seen_count >= 3) is promotion-eligible via /create-skill diagnose test-auto-fix. Lessons run through LoreKit's memory.* tools (the lorekit-memory skill); if LoreKit is not connected the loop is a silent no-op. Full contract: rules/self-improvement-loop.md.

Beyond its own lessons, it also reads the shared codebase-knowledge signal at Phase 2 for the files a failure touches (advisory, raise-care) and writes back a verified structural fact — a real product defect fixed in source at a SHA, or a repeatedly-red test file as a hotspot — under the multi-writer contract (../../../agents/shared/rules/codebase-knowledge.md; mechanics in the loop file's cross-bucket read/write sections).

Definition of done

The run is done when ANY of the following is true:

• All surface tests are green AND the structured exit summary has been printed.
• All remaining failures are prod-bug or unsure (escalated; user owns next step).
• The confidence gate scored < 80 and the fix was not written.
• A regression was detected, reverted, and the user owns the next step.
• --max-iterations (default 5) was reached.