Running and Debugging Tests
Execute and automatically debug test suites to validate behavioral correctness.
What you should do
CRITICAL TEST INTEGRITY: This workflow NEVER bypasses test failures to achieve user goals. Tests exist to improve code quality and prevent regressions—they are not obstacles to be removed when inconvenient. Only fix tests when intended behavior changes, never to accommodate bugs.
Gather test configuration – Determine the test command (e.g., npm test, pytest, go test, dotnet test) and locate the test directories or files. Optionally identify tooling for coverage (e.g., NYC, pytest --cov, go test -cover).
Execute test suite – Run the full test suite in quiet or minimal output mode. Capture the results for parsing.
[test-runner-command] [optional flags]
Example: npm test --silent, pytest -q, go test ./... -v, dotnet test --nologo
Evaluate test results –
- ✅ If all tests pass, proceed to Step 4.
- ❌ If any tests fail, proceed directly to Step 5.
Verify test quality and behavior coverage – Focus on whether tests validate meaningful behavior rather than achieving coverage percentages.
[coverage-command] # Optional for information only
Example: npx nyc --reporter=text npm test, pytest --cov=src --cov-report=term-missing, go test -coverprofile=coverage.out
- Quality indicators to check:
- Tests use behavioral names (
test_should_X_when_Y, test_rejects_X_when_Y)
- Tests verify outcomes, not implementation details
- Error paths and edge cases are tested
- Mocking is minimal and strategic (external dependencies only)
- If behavior coverage is insufficient, create new tests that validate actual user-facing functionality.
Diagnose test failures – Parse the output logs to identify failing test cases. Extract:
- Test name or function
- Source file and line
- Error message and exception type
- Stack trace or diff output
5a. Parallel resolution option – For multiple test failures (3+ failures), consider spawning parallel resolution agents:
- Assign each agent a specific test failure to resolve in isolation
- Maximum parallelization: one agent per failing test (up to system limits)
- Agents work independently on different files/functions to avoid conflicts
- Monitor completion and merge fixes before proceeding to step 11
- Fallback: If parallel execution is not available, proceed sequentially with steps 6-10
Retry on failure – If the test suite fails, rerun with -v for verbose output. Log the result to logs/test-failure.log.
Re-run failing test in isolation – Execute each failing test individually to isolate behavior.
[test-runner] path/to/file [test-selector]
Example: npm test -- test/file.test.js -t "should render correctly", pytest path/to/test_file.py::test_name -q, go test -run TestFunctionName, dotnet test --filter "TestName"
Classify failure cause –
- ✅ If failure is due to stale test logic (e.g., broken mocks, outdated assertions), proceed to Step 8a.
- ❌ If caused by source implementation bug, proceed to Step 8b.
Apply fix based on classification
- 8a. Update invalid test – Modify test code: adjust mocks, inputs, or expected outputs to match intended behavior. Ensure proper tagging and descriptive assertions.
- 8b. Patch code defect – Identify root cause and apply a minimal fix to the implementation. Maintain logic intent and regression safety.
Verify individual fix – Re-run the isolated test.
[test-runner] path/to/file [test-selector]
- If the test fails again, loop back to Step 6.
- If it passes, proceed to Step 11.
Re-run full test suite – Ensure all tests pass and no regressions were introduced.
[test-runner-command]
- If new failures occur, repeat Steps 5–10 for each.
Finalize and commit changes – Once the suite passes and coverage is adequate:
- Commit with message:
"Fix test failures and verify coverage compliance"
- Optionally run or validate the CI/CD pipeline.
- Recommend tagging a patch release if this resolves a bug cycle.
Language-Specific Guidance
- Python (PyTest):
- Full suite:
pytest -q
- Coverage:
pytest --cov=src --cov-report=term-missing
- Markers by domain:
pytest -m unit, pytest -m feature
- Isolate a test:
pytest path/to/test_file.py::test_name -q
- Verbose retry:
pytest -v
Test quality standards
Behavioral focus: Write tests with descriptive names like test_should_X_when_Y that verify observable behavior, not implementation details.
Strategic mocking: Mock external dependencies only (databases, APIs, file systems) with maximum 5 mocks per test and 3:1 mock-to-assertion ratio.
Quality over coverage: Focus on meaningful tests that would fail if functionality broke, avoid vanity tests written solely for coverage metrics.
Failure resolution: Address root causes systematically. For multiple failures, resolve in parallel when possible to maximize development velocity.
1---2name: running-and-debugging-tests3description: Executes language-agnostic test suites to validate behavioral correctness, enforce regression safety, and automatically debug or repair test failures. Includes fallback loops, coverage validation, and failure classification to maximize suite reliability. Use to validate code changes proactively, or when the user mentions running tests, test failures, or test debugging.4---5
6# Running and Debugging Tests
7
8Execute and automatically debug test suites to validate behavioral correctness.
9
10## What you should do
11
12**CRITICAL TEST INTEGRITY**: This workflow NEVER bypasses test failures to achieve user goals. Tests exist to improve code quality and prevent regressions—they are not obstacles to be removed when inconvenient. Only fix tests when intended behavior changes, never to accommodate bugs.
13
141. **Gather test configuration** – Determine the test command (e.g., `npm test`, `pytest`, `go test`, `dotnet test`) and locate the test directories or files. Optionally identify tooling for coverage (e.g., NYC, `pytest --cov`, `go test -cover`).
15
162. **Execute test suite** – Run the full test suite in quiet or minimal output mode. Capture the results for parsing.
17
18 ```bash
19 [test-runner-command] [optional flags]
20 ```
21
22 *Example:* `npm test --silent`, `pytest -q`, `go test ./... -v`, `dotnet test --nologo`
23
243. **Evaluate test results** –
25
26 * ✅ If **all tests pass**, proceed to Step 4.
27 * ❌ If **any tests fail**, proceed directly to Step 5.
28
294. **Verify test quality and behavior coverage** – Focus on whether tests validate meaningful behavior rather than achieving coverage percentages.
30
31 ```bash
32 [coverage-command] # Optional for information only
33 ```
34
35 *Example:* `npx nyc --reporter=text npm test`, `pytest --cov=src --cov-report=term-missing`, `go test -coverprofile=coverage.out`
36
37 * **Quality indicators to check:**
38 - Tests use behavioral names (`test_should_X_when_Y`, `test_rejects_X_when_Y`)
39 - Tests verify outcomes, not implementation details
40 - Error paths and edge cases are tested
41 - Mocking is minimal and strategic (external dependencies only)
42 * If behavior coverage is insufficient, create new tests that validate actual user-facing functionality.
43
445. **Diagnose test failures** – Parse the output logs to identify failing test cases. Extract:
45
46 * Test name or function
47 * Source file and line
48 * Error message and exception type
49 * Stack trace or diff output
50
515a. **Parallel resolution option** – For multiple test failures (3+ failures), consider spawning parallel resolution agents:
52
53 * Assign each agent a specific test failure to resolve in isolation
54 * Maximum parallelization: one agent per failing test (up to system limits)
55 * Agents work independently on different files/functions to avoid conflicts
56 * Monitor completion and merge fixes before proceeding to step 11
57 * **Fallback**: If parallel execution is not available, proceed sequentially with steps 6-10
58
596. **Retry on failure** – If the test suite fails, rerun with `-v` for verbose output. Log the result to `logs/test-failure.log`.
60
617. **Re-run failing test in isolation** – Execute each failing test individually to isolate behavior.
62
63 ```bash
64 [test-runner] path/to/file [test-selector]
65 ```
66
67 *Example:* `npm test -- test/file.test.js -t "should render correctly"`, `pytest path/to/test_file.py::test_name -q`, `go test -run TestFunctionName`, `dotnet test --filter "TestName"`
68
698. **Classify failure cause** –
70
71 * ✅ If failure is due to **stale test logic** (e.g., broken mocks, outdated assertions), proceed to Step 8a.
72 * ❌ If caused by **source implementation bug**, proceed to Step 8b.
73
749. **Apply fix based on classification**
75
76 * **8a. Update invalid test** – Modify test code: adjust mocks, inputs, or expected outputs to match intended behavior. Ensure proper tagging and descriptive assertions.
77 * **8b. Patch code defect** – Identify root cause and apply a minimal fix to the implementation. Maintain logic intent and regression safety.
78
7910. **Verify individual fix** – Re-run the isolated test.
80
81 ```bash
82 [test-runner] path/to/file [test-selector]
83 ```
84
85* If the test fails again, loop back to Step 6.
86* If it passes, proceed to Step 11.
87
8811. **Re-run full test suite** – Ensure all tests pass and no regressions were introduced.
89
90 ```bash
91 [test-runner-command]
92 ```
93
94 * If new failures occur, repeat Steps 5–10 for each.
95
9612. **Finalize and commit changes** – Once the suite passes and coverage is adequate:
97
98 * Commit with message: `"Fix test failures and verify coverage compliance"`
99 * Optionally run or validate the CI/CD pipeline.
100 * Recommend tagging a patch release if this resolves a bug cycle.
101
102## Language-Specific Guidance
103
104- Python (PyTest):
105 - Full suite: `pytest -q`
106 - Coverage: `pytest --cov=src --cov-report=term-missing`
107 - Markers by domain: `pytest -m unit`, `pytest -m feature`
108 - Isolate a test: `pytest path/to/test_file.py::test_name -q`
109 - Verbose retry: `pytest -v`
110
111## Test quality standards
112
113**Behavioral focus**: Write tests with descriptive names like `test_should_X_when_Y` that verify observable behavior, not implementation details.
114
115**Strategic mocking**: Mock external dependencies only (databases, APIs, file systems) with maximum 5 mocks per test and 3:1 mock-to-assertion ratio.
116
117**Quality over coverage**: Focus on meaningful tests that would fail if functionality broke, avoid vanity tests written solely for coverage metrics.
118
119**Failure resolution**: Address root causes systematically. For multiple failures, resolve in parallel when possible to maximize development velocity.
majiayu000