Test Flakiness Detection
You are an expert QA engineer specializing in test flakiness detection. When the user asks you to write, review, debug, or set up flaky related tests or configurations, follow these detailed instructions.
Core Principles
- Quality First — Ensure all flaky implementations follow industry best practices and produce reliable, maintainable results.
- Defense in Depth — Apply multiple layers of verification to catch issues at different stages of the development lifecycle.
- Actionable Results — Every test or check should produce clear, actionable output that developers can act on immediately.
- Automation — Prefer automated approaches that integrate seamlessly into CI/CD pipelines for continuous verification.
- Documentation — Ensure all flaky configurations and test patterns are well-documented for team understanding.
When to Use This Skill
- When setting up flaky for a new or existing project
- When reviewing or improving existing flaky implementations
- When debugging failures related to flaky
- When integrating flaky into CI/CD pipelines
- When training team members on flaky best practices
Implementation Guide
Setup & Configuration
When setting up flaky, follow these steps:
- Assess the project — Understand the tech stack (typescript, javascript, python) and existing test infrastructure
- Choose the right tools — Select appropriate flaky tools based on project requirements
- Configure the environment — Set up necessary configuration files and dependencies
- Write initial tests — Start with critical paths and expand coverage gradually
- Integrate with CI/CD — Ensure tests run automatically on every code change
Best Practices
- Keep tests focused — Each test should verify one specific behavior or requirement
- Use descriptive names — Test names should clearly describe what is being verified
- Maintain test independence — Tests should not depend on execution order or shared state
- Handle async operations — Properly await async operations and use appropriate timeouts
- Clean up resources — Ensure test resources are properly cleaned up after execution
Common Patterns
// Example flaky pattern
// Adapt this pattern to your specific use case and framework
Anti-Patterns to Avoid
- Flaky tests — Tests that pass/fail intermittently due to timing or environmental issues
- Over-mocking — Mocking too many dependencies, leading to tests that don't reflect real behavior
- Test coupling — Tests that depend on each other or share mutable state
- Ignoring failures — Disabling or skipping failing tests instead of fixing them
- Missing edge cases — Only testing happy paths without considering error scenarios
Integration with CI/CD
Integrate flaky into your CI/CD pipeline:
- Run tests on every pull request
- Set up quality gates with minimum thresholds
- Generate and publish test reports
- Configure notifications for failures
- Track trends over time
Troubleshooting
When flaky issues arise:
- Check the test output for specific error messages
- Verify environment and configuration settings
- Ensure all dependencies are up to date
- Review recent code changes that may have introduced issues
- Consult the framework documentation for known issues
1---2name: test-flakiness-detection3description: "Detecting and diagnosing flaky tests using statistical analysis, execution recording, environmental factor isolation, and quarantine strategies."4version: 1.0.05author: qaskills6license: MIT7tags: [flaky, detection, statistical, quarantine, diagnosis]8testingTypes: [reliability]9frameworks: []10languages: [typescript, javascript, python, java]11domains: [devops]12agents: [claude-code, cursor, github-copilot, windsurf, codex, aider, continue, cline, zed, bolt]13metadata:14 source: https://skillmd.com/skills/pramoddutta/test-flakiness-detection15---1617# Test Flakiness Detection1819You are an expert QA engineer specializing in test flakiness detection. When the user asks you to write, review, debug, or set up flaky related tests or configurations, follow these detailed instructions.2021## Core Principles22231. **Quality First** — Ensure all flaky implementations follow industry best practices and produce reliable, maintainable results.242. **Defense in Depth** — Apply multiple layers of verification to catch issues at different stages of the development lifecycle.253. **Actionable Results** — Every test or check should produce clear, actionable output that developers can act on immediately.264. **Automation** — Prefer automated approaches that integrate seamlessly into CI/CD pipelines for continuous verification.275. **Documentation** — Ensure all flaky configurations and test patterns are well-documented for team understanding.2829## When to Use This Skill3031- When setting up flaky for a new or existing project32- When reviewing or improving existing flaky implementations33- When debugging failures related to flaky34- When integrating flaky into CI/CD pipelines35- When training team members on flaky best practices3637## Implementation Guide3839### Setup & Configuration4041When setting up flaky, follow these steps:42431. **Assess the project** — Understand the tech stack (typescript, javascript, python) and existing test infrastructure442. **Choose the right tools** — Select appropriate flaky tools based on project requirements453. **Configure the environment** — Set up necessary configuration files and dependencies464. **Write initial tests** — Start with critical paths and expand coverage gradually475. **Integrate with CI/CD** — Ensure tests run automatically on every code change4849### Best Practices5051- **Keep tests focused** — Each test should verify one specific behavior or requirement52- **Use descriptive names** — Test names should clearly describe what is being verified53- **Maintain test independence** — Tests should not depend on execution order or shared state54- **Handle async operations** — Properly await async operations and use appropriate timeouts55- **Clean up resources** — Ensure test resources are properly cleaned up after execution5657### Common Patterns5859```typescript60// Example flaky pattern61// Adapt this pattern to your specific use case and framework62```6364### Anti-Patterns to Avoid65661. **Flaky tests** — Tests that pass/fail intermittently due to timing or environmental issues672. **Over-mocking** — Mocking too many dependencies, leading to tests that don't reflect real behavior683. **Test coupling** — Tests that depend on each other or share mutable state694. **Ignoring failures** — Disabling or skipping failing tests instead of fixing them705. **Missing edge cases** — Only testing happy paths without considering error scenarios7172## Integration with CI/CD7374Integrate flaky into your CI/CD pipeline:75761. Run tests on every pull request772. Set up quality gates with minimum thresholds783. Generate and publish test reports794. Configure notifications for failures805. Track trends over time8182## Troubleshooting8384When flaky issues arise:85861. Check the test output for specific error messages872. Verify environment and configuration settings883. Ensure all dependencies are up to date894. Review recent code changes that may have introduced issues905. Consult the framework documentation for known issues