ChloeVPin Lifecycle: draft
Test-Effectiveness Analysis
Purpose and scope
Use this skill to evaluate whether tests detect acceptance-relevant behavioral faults, especially in AI-generated changes. It supplements ordinary test execution with boundary cases, negative cases, differential checks, and mutation-oriented diagnostics. It does not claim that mutation score proves correctness or prescribe one universal threshold.
Triggers and prerequisites
Trigger when a change adds or modifies behavior, when tests are newly generated, when a patch passes tests but remains semantically uncertain, or when regressions recur. Prerequisites: acceptance criteria, baseline behavior, executable tests, and a way to create or obtain representative mutations.
Decision criteria
Treat a test as effective when it fails for a plausible incorrect behavior relevant to the acceptance criteria and remains behavior-specific at boundaries and negative paths. Do not accept a score or passing suite as sufficient when representative faults survive unexplored.
Procedure
- Translate the requested behavior into observable positive, negative, boundary, and invariant cases.
- Run the baseline tests and record their revision, command, exit status, and output digest.
- Inspect whether tests assert externally observable behavior rather than only execution, non-null values, or implementation details.
- Introduce representative fault mutations near the changed behavior: operator/branch changes, boundary shifts, omitted calls, altered errors, and incorrect side effects where applicable.
- Run the test suite against each mutation and classify killed, surviving, equivalent, invalid, or unexecuted mutations. Do not count invalid mutants as surviving evidence.
- Investigate surviving mutations. Add the smallest test that distinguishes the intended behavior when the mutation represents a plausible fault.
- Use differential or oracle comparison when a reference implementation exists, and check that tests cover unaffected behavior to detect regressions.
- Record mutation selection, test results, limitations, residual uncertainty, and the next review trigger. Treat mutation results as diagnostic evidence, not a correctness certificate.
Acceptance checklist
- Acceptance criteria include positive, negative, boundary, or invariant cases as applicable.
- Baseline and focused tests were run with captured evidence.
- Assertions check externally observable behavior and side effects.
- Representative mutations were selected and classified.
- Surviving plausible mutations were investigated or explicitly accepted with rationale.
- Regression behavior outside the changed path was checked.
- Mutation limitations, equivalent mutants, and residual uncertainty are recorded.
Examples and counterexamples
Good: A mutation removes a required authorization check and the relevant negative test fails, showing that the test detects the security behavior.
Bad: Count invalid or equivalent mutants as evidence that the suite is weak, or treat a mutation score as correctness proof.
Failure modes and recovery
If mutation tooling cannot run, record the exact blocker and use manually designed fault injections only as limited evidence. If a mutation is equivalent, document why rather than weakening the test. If mutation score improves while behavior coverage worsens, reject the metric as misleading and inspect the test assertions.
Validation evidence and provenance
Claims in this skill map to graded findings in docs/research.md:
- Mutants are coupled with real faults (Q1, Strong): Petrović et al. (ICSE 2021), 15 million mutants at Google.
- Mutation-augmented test writing raises coverage (Q2, Moderate): Roman & Mnich 2021 controlled experiment, small-sample limits stated.
- Coverage alone overstates test quality (Q1/Q3, supported): the proxy-metric debate is documented; mutation analysis is the stronger proxy per Q1.
Source boundary: sources support the analysis lens, not any specific coverage threshold in this skill.
Confidence: medium. Freshness: review annually and after material changes to testing tools or mutation methodology.
Disconfirmation: independent large-scale studies contradicting the mutant–fault coupling would require re-grading and possibly reversing the recommendation.
Related skills and conflicts
Related: repository-change-verification, regression-test-design, differential-patch-review, and dependency-security-audit. This skill does not override project-specific test policy or human review.