Test Fixing
Systematically identify and fix all failing tests using smart error grouping, root-cause analysis, and iterative validation. Avoid the "whack-a-mole" approach: group failures, fix root causes once, and verify each group before moving on.
When to Use
- User explicitly asks to fix tests: "fix these tests", "make tests pass", "get the suite green"
- User reports test failures: "tests are failing", "test suite is broken", "CI is red"
- Implementation is complete and the user wants tests passing before commit
- CI/CD pipeline failures caused by test regressions
- After a major refactor where multiple tests are broken simultaneously
- After dependency upgrades that may have introduced breaking changes
Prerequisites
- The project has an existing test suite (pytest, unittest, jest, etc.)
- A working test runner is available (
make test, uv run pytest, npm test, etc.)
- On Windows host (PowerShell), prefer
uv run pytest or the project's configured runner. If make is unavailable, invoke the runner directly.
- Check for a
CLAUDE.md, CONTRIBUTING.md, or project style guide before making edits. Adhere to project-specific coding standards.
- If the project uses
references/ or scripts/ directories, load them as needed (see Procedure).
Procedure
1. Initial Test Run — Identify Scope
Run the project's primary test command to capture the full failure surface.
Windows (PowerShell):
uv run pytest --tb=short -q
If a Makefile is present:
make test
Analyze the output for:
- Total number of failures vs. errors (they have different root causes).
- Common error patterns (repeated tracebacks, shared import lines).
- Affected modules, files, and specific test cases.
- Environment-related failures (missing env vars, missing config files, port conflicts).
- Skipped (
SKIPPED) or expected-fail (XFAIL) tests — investigate why before ignoring.
Capture the list of failing test node IDs for targeted re-runs:
uv run pytest --tb=line -q 2>&1 | Select-String "FAILED"
2. Smart Error Grouping
Group failures to solve root causes once rather than fixing symptoms one by one.
Group by:
- Error Type:
ImportError, ModuleNotFoundError, AttributeError, TypeError, AssertionError, TimeoutError.
- Location: Failures concentrated in a specific module, directory, or test file.
- Root Cause: API breaking changes, dependency updates, configuration drift, fixture issues, environment mismatches.
Prioritize groups by:
- Impact — Fix the error causing the most failures first.
- Dependency Order — Fix infrastructure/setup failures before business logic failures. A single
ImportError can cascade into dozens of downstream failures.
3. Fix Order Strategy
Follow this hierarchy to ensure a stable foundation before addressing higher-level logic:
Level 1 — Infrastructure & Environment
ImportError, ModuleNotFoundError
- Missing dependencies or incorrect
pyproject.toml / requirements.txt
- Environment variable or configuration mismatches
- Fixture setup/teardown failures
- Database or service connection errors
Level 2 — Interface & API Changes
- Function signature mismatches (
TypeError)
- Renamed methods or moved modules (
AttributeError)
- Updated return types or data structures (e.g.,
dict → dataclass)
- Changed CLI flags or API response shapes
Level 3 — Logic & Behavioral Issues
AssertionError (incorrect values)
- Edge case failures (empty input, boundary values, timezone issues)
- Race conditions or timeout issues
- Flaky tests (intermittent failures)
4. Systematic Fixing Process (Per Group)
For each group, starting with the highest priority:
4a. Identify Root Cause
4b. Implement Fix
- Apply targeted, minimal changes using the Edit tool.
- Adhere to project-specific coding standards (refer to
CLAUDE.md or equivalent).
- Keep changes minimal to avoid introducing new regressions.
- If a
scripts/ directory contains helper scripts (e.g., lint, format, type-check), run them after each fix to catch side effects early.
4c. Verify Fix (Iterative Validation)
Run only the affected subset of tests to save time and reduce noise.
# Run a specific test file
uv run pytest tests/path/to/test_file.py -v
# Run tests matching a specific keyword/pattern
uv run pytest -k "keyword" -v
# Run only failing tests from the last run
uv run pytest --lf -v
# Run a single test node ID
uv run pytest tests/path/to/test_file.py::TestClass::test_method -v
# Extra verbose output for deep debugging
uv run pytest tests/path/to/test_file.py -vv --tb=long
Ensure the current group passes completely before moving to the next group.
4d. Move to Next Group
Repeat 4a–4c for the next priority group.
5. Final Verification
Once all groups are addressed:
# Run the complete test suite
uv run pytest --tb=short
# Or via Makefile
make test
Pitfalls
- Do not ignore
SKIPPED or XFAIL tests without investigating why they are skipped — they may mask real failures.
- Do not modify the test suite to match incorrect behavior in the code. If the test is wrong, confirm the expected behavior first, then update the test with a clear reason.
- Do not use
pytest.mark.skip as a temporary fix without a corresponding TODO comment or ticket reference.
- Do not perform bulk "find and replace" across the codebase without verifying the impact on all affected tests.
- Do not comment out failing tests or weaken assertions just to make the suite pass.
- Do not fix symptoms individually ("whack-a-mole") — always group and address root causes.
- Do not run the full suite after every single change — use targeted subset runs to maintain a fast feedback loop.
- Do not forget to remove debug
print() statements or temporary logging added during investigation.
- Do not commit fixes for multiple unrelated groups in a single atomic commit if the project expects granular history — prefer one commit per root-cause group.
- Watch for flaky tests: if a test passes on re-run but failed initially, investigate race conditions, time-dependent logic, or shared mutable state before dismissing it.
- Watch for environment drift: failures that only appear in CI but not locally often indicate missing env vars, different Python/Node versions, or OS-specific path handling (especially Windows vs. Linux path separators).
Verification
Confirm each item before declaring the task complete:
Examples
Scenario: User reports "The tests are failing after my refactor."
- Run:
uv run pytest --tb=short -q → 15 failures.
- Group:
- 8 ×
ImportError → Root: utils.py moved to core/utils.py.
- 5 ×
AttributeError → Root: get_user() now returns a User object instead of a dict.
- 2 ×
AssertionError → Root: Logic bug in date calculation.
- Fix Group 1 (Infrastructure): Update imports across affected files → Run
uv run pytest tests/test_utils.py -v → Pass.
- Fix Group 2 (Interface): Update call sites to access object attributes (
user.name instead of user["name"]) → Run uv run pytest tests/test_users.py -v → Pass.
- Fix Group 3 (Logic): Correct date calculation edge case → Run
uv run pytest tests/test_dates.py -v → Pass.
- Final: Run
uv run pytest --tb=short → All pass ✓.
Related Skills
debugging — General debugging workflows for non-test-specific failures.
code-review — Reviewing fixes before commit to ensure minimal, standards-compliant changes.
refactoring — Safe refactoring practices that minimize test breakage.
1---
2name: test-fixing
3version: 1.1.1
4description: "Groups failing suite errors by type and root cause, then repairs infrastructure, API drift, and assertion bugs until the project's runner is green. Use when tests fail, CI is red, or a refactor broke the suite. Do not use to author a new pytest or TestNG tree, or to skip or weaken assertions for a pass."
5risk: safe
6source: openrouter-deepsearch
7date_added: 2026-06-14
8metadata:
9 source: https://skillmd.com/skills/kayforkind/test-fixing
10---
11
12# Test Fixing
13
14Systematically identify and fix all failing tests using smart error grouping, root-cause analysis, and iterative validation. Avoid the "whack-a-mole" approach: group failures, fix root causes once, and verify each group before moving on.
15
16## When to Use
17
18- User explicitly asks to fix tests: "fix these tests", "make tests pass", "get the suite green"
19- User reports test failures: "tests are failing", "test suite is broken", "CI is red"
20- Implementation is complete and the user wants tests passing before commit
21- CI/CD pipeline failures caused by test regressions
22- After a major refactor where multiple tests are broken simultaneously
23- After dependency upgrades that may have introduced breaking changes
24
25## Prerequisites
26
27- The project has an existing test suite (pytest, unittest, jest, etc.)
28- A working test runner is available (`make test`, `uv run pytest`, `npm test`, etc.)
29- On Windows host (PowerShell), prefer `uv run pytest` or the project's configured runner. If `make` is unavailable, invoke the runner directly.
30- Check for a `CLAUDE.md`, `CONTRIBUTING.md`, or project style guide before making edits. Adhere to project-specific coding standards.
31- If the project uses `references/` or `scripts/` directories, load them as needed (see Procedure).
32
33## Procedure
34
35### 1. Initial Test Run — Identify Scope
36
37Run the project's primary test command to capture the full failure surface.
38
39**Windows (PowerShell):**
40```powershell
41uv run pytest --tb=short -q
42```
43
44**If a Makefile is present:**
45```powershell
46make test
47```
48
49Analyze the output for:
50- Total number of **failures** vs. **errors** (they have different root causes).
51- Common error patterns (repeated tracebacks, shared import lines).
52- Affected modules, files, and specific test cases.
53- Environment-related failures (missing env vars, missing config files, port conflicts).
54- Skipped (`SKIPPED`) or expected-fail (`XFAIL`) tests — investigate why before ignoring.
55
56Capture the list of failing test node IDs for targeted re-runs:
57```powershell
58uv run pytest --tb=line -q 2>&1 | Select-String "FAILED"
59```
60
61### 2. Smart Error Grouping
62
63Group failures to solve root causes once rather than fixing symptoms one by one.
64
65**Group by:**
66- **Error Type**: `ImportError`, `ModuleNotFoundError`, `AttributeError`, `TypeError`, `AssertionError`, `TimeoutError`.
67- **Location**: Failures concentrated in a specific module, directory, or test file.
68- **Root Cause**: API breaking changes, dependency updates, configuration drift, fixture issues, environment mismatches.
69
70**Prioritize groups by:**
711. **Impact** — Fix the error causing the most failures first.
722. **Dependency Order** — Fix infrastructure/setup failures before business logic failures. A single `ImportError` can cascade into dozens of downstream failures.
73
74### 3. Fix Order Strategy
75
76Follow this hierarchy to ensure a stable foundation before addressing higher-level logic:
77
78**Level 1 — Infrastructure & Environment**
79- `ImportError`, `ModuleNotFoundError`
80- Missing dependencies or incorrect `pyproject.toml` / `requirements.txt`
81- Environment variable or configuration mismatches
82- Fixture setup/teardown failures
83- Database or service connection errors
84
85**Level 2 — Interface & API Changes**
86- Function signature mismatches (`TypeError`)
87- Renamed methods or moved modules (`AttributeError`)
88- Updated return types or data structures (e.g., `dict` → dataclass)
89- Changed CLI flags or API response shapes
90
91**Level 3 — Logic & Behavioral Issues**
92- `AssertionError` (incorrect values)
93- Edge case failures (empty input, boundary values, timezone issues)
94- Race conditions or timeout issues
95- Flaky tests (intermittent failures)
96
97### 4. Systematic Fixing Process (Per Group)
98
99For each group, starting with the highest priority:
100
101#### 4a. Identify Root Cause
102- Read the traceback from the **bottom up** — the last frame usually points to the failing line.
103- Correlate failures with recent changes:
104 ```powershell
105 git diff HEAD~5 --name-only
106 git log --oneline -10
107 ```
108- Inspect the failing line of code and the corresponding test assertion side by side.
109- If the project has a `references/` directory with architecture or API docs, load the relevant reference file to understand expected behavior before editing.
110
111#### 4b. Implement Fix
112- Apply targeted, minimal changes using the Edit tool.
113- Adhere to project-specific coding standards (refer to `CLAUDE.md` or equivalent).
114- Keep changes minimal to avoid introducing new regressions.
115- If a `scripts/` directory contains helper scripts (e.g., lint, format, type-check), run them after each fix to catch side effects early.
116
117#### 4c. Verify Fix (Iterative Validation)
118Run only the affected subset of tests to save time and reduce noise.
119
120```powershell
121# Run a specific test file
122uv run pytest tests/path/to/test_file.py -v
123
124# Run tests matching a specific keyword/pattern
125uv run pytest -k "keyword" -v
126
127# Run only failing tests from the last run
128uv run pytest --lf -v
129
130# Run a single test node ID
131uv run pytest tests/path/to/test_file.py::TestClass::test_method -v
132
133# Extra verbose output for deep debugging
134uv run pytest tests/path/to/test_file.py -vv --tb=long
135```
136
137Ensure the current group passes **completely** before moving to the next group.
138
139#### 4d. Move to Next Group
140Repeat 4a–4c for the next priority group.
141
142### 5. Final Verification
143
144Once all groups are addressed:
145
146```powershell
147# Run the complete test suite
148uv run pytest --tb=short
149
150# Or via Makefile
151make test
152```
153
154- Verify that **no new regressions** were introduced in previously passing modules.
155- Check coverage reports if available to ensure no critical paths were accidentally skipped:
156 ```powershell
157 uv run pytest --cov=src --cov-report=term-missing
158 ```
159- Remove all temporary debug prints/logs added during investigation.
160- Run linters and formatters if configured:
161 ```powershell
162 uv run ruff check . --fix
163 uv run ruff format .
164 ```
165
166## Pitfalls
167
168- **Do not** ignore `SKIPPED` or `XFAIL` tests without investigating why they are skipped — they may mask real failures.
169- **Do not** modify the test suite to match incorrect behavior in the code. If the test is wrong, confirm the expected behavior first, then update the test with a clear reason.
170- **Do not** use `pytest.mark.skip` as a temporary fix without a corresponding `TODO` comment or ticket reference.
171- **Do not** perform bulk "find and replace" across the codebase without verifying the impact on all affected tests.
172- **Do not** comment out failing tests or weaken assertions just to make the suite pass.
173- **Do not** fix symptoms individually ("whack-a-mole") — always group and address root causes.
174- **Do not** run the full suite after every single change — use targeted subset runs to maintain a fast feedback loop.
175- **Do not** forget to remove debug `print()` statements or temporary logging added during investigation.
176- **Do not** commit fixes for multiple unrelated groups in a single atomic commit if the project expects granular history — prefer one commit per root-cause group.
177- **Watch for flaky tests**: if a test passes on re-run but failed initially, investigate race conditions, time-dependent logic, or shared mutable state before dismissing it.
178- **Watch for environment drift**: failures that only appear in CI but not locally often indicate missing env vars, different Python/Node versions, or OS-specific path handling (especially Windows vs. Linux path separators).
179
180## Verification
181
182Confirm each item before declaring the task complete:
183
184- [ ] All tests in each failing group pass individually.
185- [ ] The full test suite runs without errors:
186 ```powershell
187 uv run pytest --tb=short
188 ```
189- [ ] No new regressions introduced in previously passing modules.
190- [ ] Code adheres to the project's style guide and architecture (`CLAUDE.md` / `CONTRIBUTING.md`).
191- [ ] All temporary debug prints/logs have been removed.
192- [ ] Linters and formatters pass (if configured).
193- [ ] `SKIPPED` and `XFAIL` tests have been investigated and documented if still skipped.
194- [ ] Coverage has not dropped on critical paths (if coverage is tracked).
195
196## Examples
197
198**Scenario**: User reports "The tests are failing after my refactor."
199
2001. **Run**: `uv run pytest --tb=short -q` → 15 failures.
2012. **Group**:
202 - 8 × `ImportError` → Root: `utils.py` moved to `core/utils.py`.
203 - 5 × `AttributeError` → Root: `get_user()` now returns a `User` object instead of a `dict`.
204 - 2 × `AssertionError` → Root: Logic bug in date calculation.
2053. **Fix Group 1** (Infrastructure): Update imports across affected files → Run `uv run pytest tests/test_utils.py -v` → Pass.
2064. **Fix Group 2** (Interface): Update call sites to access object attributes (`user.name` instead of `user["name"]`) → Run `uv run pytest tests/test_users.py -v` → Pass.
2075. **Fix Group 3** (Logic): Correct date calculation edge case → Run `uv run pytest tests/test_dates.py -v` → Pass.
2086. **Final**: Run `uv run pytest --tb=short` → All pass ✓.
209
210## Related Skills
211
212- `debugging` — General debugging workflows for non-test-specific failures.
213- `code-review` — Reviewing fixes before commit to ensure minimal, standards-compliant changes.
214- `refactoring` — Safe refactoring practices that minimize test breakage.