# Test Confidence

> AI-driven test execution. Opus decides what to run and how confident to be, based on your diff.

- Skill: `antiwork/test-confidence` (Agent Skill)
- Install (CLI): `npx skillmds@latest add antiwork/test-confidence`
- Raw SKILL.md: https://api.skillmd.com/api/skills/antiwork/test-confidence/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: antiwork (https://skillmd.com/u/antiwork)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/antiwork/test-confidence

---


# Test Confidence

Run `bin/test-confidence` to have Opus 4.7 analyze your diff, decide the risk level, plan which tests to run and in what order, and set confidence milestones. The AI decides the shape of the curve based on this specific diff.

## Usage

```bash
bin/test-confidence            # Run to 99%, stop. Skips past pre-existing failures.
bin/test-confidence --full     # Run to 100%
bin/test-confidence --strict   # Halt on any failure, including pre-existing
```

`ANTHROPIC_API_KEY` is auto-sourced from `.env` if not exported.

If `$ARGUMENTS` is provided, pass it through: `bin/test-confidence $ARGUMENTS`

## How it works

1. Finds changed files (branch diff vs main for PR branches, local diff on main)
2. Hashes the diff and checks `tmp/test-confidence/` for a cached plan; reuses if found
3. Otherwise sends the diff + spec tree + touched directories to Opus 4.7 in one call
4. Opus returns a plan: risk level, ordered test list, confidence milestones
5. Script executes the plan, showing yellow progress bar toward 99%
6. At 99%, bar turns green. Safe to commit.
7. With `--full`, continues running remaining tests toward 100%

## Pre-existing failure detection

When a spec fails, the script applies a two-step check:

1. **Path heuristic** — does the failing spec file or its source counterpart (e.g., `app/models/user.rb` for `spec/models/user_spec.rb`) appear in the diff?
   - **In the diff** → real regression. Halt immediately.
   - **Not in the diff** → suspect pre-existing. Verify on merge-base.
2. **Merge-base verify** — re-run the failing rspec examples in a temporary worktree at the branch's merge-base with `main`.
   - **Still fails on merge-base** → confirmed pre-existing. Continue.
   - **Passes on merge-base** → cross-file regression caught. Halt.

The verify step is skipped (heuristic verdict trusted) when the diff includes a `db/migrate/*.rb` (DB schema drift) or `Gemfile.lock` (bundler drift), or when no merge-base is available. `--strict` skips both checks and halts on every failure.

Cost profile: zero overhead in the common no-failure case; verify only runs on heuristic-flagged "pre-existing" failures. Catches the dangerous direction (silent regression skipped) while keeping false-alarm investigation costs bounded.

The key insight: Opus decides ad hoc how many tests are needed for each confidence level. A comment-only change might need 2 tests for 99%. A payment model refactor might need 100.

## When to use

Run this before every commit. It replaces manually picking which specs to run.

$ARGUMENTS