Jamie-BitFlight Analyze Test Failures
Analyze failing test cases with a balanced, investigative approach.
Context
Consult ../python3-development/references/python3-standards.md when shared testing or quality rules from this plugin apply; full standards, graphs, and amendment process are documented there.
When tests fail, there are two primary possibilities:
- False positive: The test itself is incorrect
- True positive: The test discovered a genuine bug
Assuming tests are wrong by default is a dangerous anti-pattern that defeats the purpose of testing.
Analysis Process
1. Initial Analysis
- Read the failing test carefully, understanding its intent
- Examine the test's assertions and expected behavior
- Review the error message and stack trace
2. Investigate the Implementation
- Check the actual implementation being tested
- Trace through the code path that leads to the failure
- Verify that implementation matches documented behavior
3. Apply Critical Thinking
For each failing test, ask:
- What behavior is the test trying to verify?
- Is this behavior clearly documented or implied by the API design?
- Does the current implementation actually provide this behavior?
- Could this be an edge case the implementation missed?
4. Make a Determination
Classify the failure as one of:
| Classification | Meaning |
|---|---|
| Test Bug | Test's expectations are incorrect |
| Implementation Bug | Code doesn't behave as it should |
| Ambiguous | Intended behavior is unclear |
5. Document Reasoning
Provide clear explanation including:
- Evidence supporting the conclusion
- Specific mismatch between expectation and reality
- Recommended fix (to test or implementation)
Example Analyses
Example 1: Ambiguous Behavior
Scenario: Test expects calculateDiscount(100, 0.2) to return 20, but it returns 80
Analysis:
- Test assumes function returns discount amount
- Implementation returns price after discount
- Function name is ambiguous
Determination: Ambiguous Recommendation: Check documentation or clarify intended behavior
Example 2: Implementation Bug
Scenario: Test expects validateEmail("user@example.com") to return true, but it returns false
Analysis:
- Test provides a valid email format
- Implementation regex is missing support for dots in domain
- Other valid emails also fail
Determination: Implementation Bug Recommendation: Fix the regex to properly validate email addresses per RFC standards
Example 3: Test Bug
Scenario: Test expects divide(10, 0) to return 0, but it throws an error
Analysis:
- Test assumes division by zero returns 0
- Implementation throws DivisionByZeroError
- Standard mathematical behavior is to treat as undefined/error
Determination: Test Bug Recommendation: Update test to expect an error, not 0
Output Format
For each failing test, provide:
Test: [test name/description]
Failure: [what failed and how]
Investigation:
- Test expects: [expected behavior]
- Implementation does: [actual behavior]
- Root cause: [why they differ]
Determination: [Test Bug | Implementation Bug | Ambiguous]
Recommendation:
[Specific fix to either test or implementation]
Key Principles
- NEVER automatically assume the test is wrong
- ALWAYS consider that the test might have found a real bug
- When uncertain, lean toward investigating the implementation
- Tests are often your specification - they define expected behavior
- A failing test is a gift - it's either catching a bug or clarifying requirements
Related Skills
- test-failure-mindset: Set investigative approach for session
- comprehensive-test-review: Full test suite review