Test Results Analyzer

Test Results Analyzer

Especialista en extraer insights de resultados de tests. Identifica patterns, reduce flakiness, y mejora la salud general del test suite.

Cuándo Usar Este Skill

• Analizar test reports y failures
• Identificar flaky tests
• Detectar patterns de regresión
• Mejorar test suite health
• Reducir tiempos de CI/CD
• Priorizar tests por impacto

Test Suite Health Metrics

RELIABILITY:
- Pass rate: >99% (non-flaky)
- Flaky rate: <1%
- False positive rate: <0.1%

SPEED:
- Unit tests: <5 min total
- Integration: <15 min total
- E2E: <30 min total

COVERAGE:
- Line coverage: >80%
- Branch coverage: >70%
- Critical path: 100%

MAINTENANCE:
- Tests per file ratio
- Avg test age
- Update frequency

Failure Analysis Framework

CUANDO UN TEST FALLA:

1. CATEGORIZE
   - Real bug (test caught issue)
   - Flaky (inconsistent)
   - Environment (infra issue)
   - Test bug (test is wrong)

2. INVESTIGATE
   - Check error message
   - Review recent changes
   - Check CI environment
   - Try local reproduction

3. RESOLVE
   - Real bug → fix code
   - Flaky → fix or quarantine
   - Environment → fix infra
   - Test bug → fix test

4. PREVENT
   - Add to monitoring
   - Update test patterns
   - Share learnings

Flaky Test Detection

SIGNS OF FLAKINESS:
- Passes locally, fails in CI
- Fails intermittently
- Time-dependent failures
- Order-dependent failures
- Resource-dependent failures

COMMON CAUSES:
- Timing/race conditions
- Shared state between tests
- External dependencies
- Hardcoded values (dates, IDs)
- Non-deterministic order

SOLUTIONS:
- Add explicit waits
- Isolate test state
- Mock external services
- Use relative/dynamic values
- Ensure test independence

Test Report Template

## Test Results: [Build/PR]

**Date:** [date]
**Total Tests:** X
**Duration:** Y minutes

### Summary
| Status | Count | % |
|--------|-------|---|
| ✅ Passed | X | Y% |
| ❌ Failed | X | Y% |
| ⏭️ Skipped | X | Y% |

### Failed Tests
| Test | Error | Category | Action |
|------|-------|----------|--------|
| test_user_login | Timeout | Flaky | Quarantine |
| test_payment | Assert | Real bug | Fix |

### Flaky Tests (Last 7 Days)
| Test | Fail Rate | Last Fail | Status |
|------|-----------|-----------|--------|
| test_api_timeout | 15% | Today | Investigating |
| test_race_condition | 5% | 3 days ago | Fixed |

### Coverage
- Lines: X%
- Branches: Y%
- Functions: Z%

### Trends
- Pass rate: [trend chart]
- Duration: [trend chart]

Pattern Recognition

LOOK FOR PATTERNS:

BY TIME:
- Failures at specific times?
- Day of week patterns?
- After deployments?

BY TYPE:
- Same error message?
- Same module/area?
- Same test category?

BY CHANGE:
- After specific commits?
- New dependencies?
- Infrastructure changes?

BY ENVIRONMENT:
- Specific CI runner?
- Browser/OS combination?
- Resource constraints?

Test Optimization

REDUCE CI TIME:

1. PARALLELIZE
   - Split by type
   - Shard large suites
   - Run independent in parallel

2. PRIORITIZE
   - Run fast tests first
   - Critical paths early
   - Flaky tests separately

3. CACHE
   - Dependencies
   - Build artifacts
   - Test data

4. SKIP SMART
   - Affected tests only
   - Skip unchanged areas
   - Conditional on file changes

5. OPTIMIZE TESTS
   - Remove redundant tests
   - Merge similar tests
   - Use lighter setups

Test Health Dashboard

## Test Suite Health

### Overall Status: 🟢 Healthy

### Key Metrics
| Metric | Current | Target | Trend |
|--------|---------|--------|-------|
| Pass Rate | 99.2% | >99% | ↑ |
| Flaky Rate | 0.8% | <1% | ↓ |
| Avg Duration | 12 min | <15 min | → |
| Coverage | 82% | >80% | ↑ |

### Problem Areas
1. **auth/* tests**: 3 flaky tests - assigned to @person
2. **api/* tests**: Slow (5 min) - optimization needed

### Recent Improvements
- Fixed 5 flaky tests last week
- Reduced E2E time by 20%
- Added coverage for payments module

### Action Items
- [ ] Investigate timeout issues in integration tests
- [ ] Add retry logic for network tests
- [ ] Split large test file (user_test.py)

Quarantine Process

WHEN TO QUARANTINE:
- Flaky rate >10%
- Blocks CI consistently
- No quick fix available

QUARANTINE PROCESS:
1. Move to quarantine suite
2. Log issue with details
3. Assign owner
4. Set review date
5. Run separately (nightly)

RELEASE FROM QUARANTINE:
1. Root cause fixed
2. Passes 10x consecutive runs
3. No flakiness in 7 days
4. Move back to main suite

Test Categorization

BY TYPE:
- Unit: Fast, isolated, no I/O
- Integration: Multiple components
- E2E: Full user flows
- Performance: Load/speed

BY PRIORITY:
- P0: Critical paths, run always
- P1: Important, run on PR
- P2: Nice to have, run nightly
- P3: Edge cases, run weekly

BY STABILITY:
- Stable: Always passes/fails consistently
- Flaky: Intermittent failures
- Quarantined: Known issues

CI/CD Integration

# Test job with analysis
test:
  runs-on: ubuntu-latest
  steps:
    - name: Run tests
      run: npm test -- --json > results.json
      
    - name: Analyze results
      run: |
        node analyze-results.js results.json
        
    - name: Upload report
      uses: actions/upload-artifact@v3
      with:
        name: test-report
        path: report.html
        
    - name: Fail on flaky
      if: env.FLAKY_COUNT > 0
      run: exit 1

Mejores Prácticas

1. Fix flaky immediately - They erode trust
2. Track trends - Single failures matter less than patterns
3. Categorize failures - Not all failures are equal
4. Automate analysis - Manual review doesn't scale
5. Set quality gates - Block on test health metrics
6. Review regularly - Test suites need maintenance

Filosofía

"A test suite is only as valuable as its reliability. Flaky tests are worse than no tests—they teach the team to ignore failures."

El objetivo es mantener un test suite que el equipo confía, que falla cuando hay problemas reales, y pasa cuando todo está bien.