Flaky Detection

Flaky Test Detection

This skill identifies and analyzes flaky tests - tests that pass and fail intermittently without code changes.

When to Use

• Test fails unexpectedly after passing previously
• User asks "is this test flaky?"
• User asks "why does this test fail sometimes?"
• Test results are inconsistent across runs
• User wants to identify unreliable tests

How It Works

1. Query test execution history for the failing test
2. Analyze pass/fail patterns over time
3. Identify flakiness indicators:
   • Timing-dependent failures
   • Race conditions
   • External dependencies
   • Resource contention
4. Calculate flakiness score
5. Suggest remediation strategies

Implementation

Use the following Argus MCP tools:

1. argus_quality_stats - Get test quality metrics and flakiness scores

   • Input: { "project_id": "<project>", "test_id": "<test>" }
   • Returns: Flakiness score, failure patterns, execution history

2. argus_healing_patterns - Find common fix patterns for flaky tests

   • Input: { "failure_type": "flaky", "test_id": "<test>" }
   • Returns: Suggested fixes based on similar resolved issues

Output Format

Flakiness Analysis

Flakiness Indicators

• Timing Issues: Test depends on async operations without proper waits
• Race Conditions: Multiple threads/processes competing for resources
• External Dependencies: Network, databases, or third-party services
• State Leakage: Previous tests affecting current test state

Remediation Steps

1. Add explicit waits/retries for async operations
2. Isolate test state with fixtures
3. Mock external dependencies
4. Use deterministic test data

Converted and distributed by TomeVault — claim your Tome and manage your conversions.