Automated Test Case

Regression Test Mining

Build the smallest maintainable test that demonstrably catches the largest important defect cluster. Treat historical coverage as an empirical claim, not a similarity judgment.

Definition Of Success

The primary success metric is historical application defects proven caught / eligible historical defects reviewed.

A historical code defect counts as caught only when the same unchanged test:

1. executes the real affected application path against the exact pre-fix revision, or a faithful inverse of the documented historical fix;
2. fails for the historically correct reason; and
3. passes against the corrected application code.

Model mutants, simulated state machines, synthetic contradictions, assertion totals, scenario counts, and passing current-code tests do not count as historical catches. They may support design or generalization, but report them separately as hypothesis, model coverage, or data-integrity coverage.

Do not call a suite successful merely because it has many checks. If no historical application defect was executed and rejected, its proven historical catch count is zero.

A proposed broad regression test is good only when the same maintainable test proves catches across multiple distinct historical defects in its selected cluster, passes the corrected code and valid data/configuration, runs deterministically before production, and reports exactly what remains uncovered. Otherwise classify it as a single-defect regression, a model hypothesis, or an integrity check.

Workflow

1. Establish the issue universe from the complete available tracker export, comments, release history, support evidence, and code history. Prefer an authenticated tracker API or complete JSON export when available. Review every eligible issue and its comments, not a GUI sample. Work read-only until implementation is authorized, and remove run-created exports after analysis.
2. Review issues individually before clustering them. Record the actual failure, confirmed cause when known, reproducibility, affected subsystem, severity, fix revision, and whether the issue was code, configuration, data, or an unimplemented requirement.
3. Cluster by shared failure mechanism, not matching words or screens. For each cluster, hypothesize the smallest invariant or scenario generator that could have caught the most issues. Do not merge configuration behaving correctly, corrupt one-off records, and code regressions into one claimed test family.
4. Rank candidate tests first by proven historical application catches, then severity prevented, execution cost, determinism, maintenance cost, and ability to run before production. State the exact scenario and assertions before implementation.
5. Choose the lowest useful layer:
   • unit tests for pure rules and calculations;
   • integration tests for persistence, routing, queues, permissions, and cross-module contracts;
   • browser tests only for behavior that depends on rendered state or interaction;
   • data-integrity tests for malformed, contradictory, or migrated records.
6. Keep a fast CLI path suitable for developer use and CI. A GUI may explain or configure scenarios, but it must invoke the same approved test definitions and never become the only runner.
7. Generate temporary fixtures with unmistakable markers, isolate them from production, and always verify cleanup. Never use production data or mutate a production system to prove a test.
8. Verify the negative and positive controls against application code. Run the same unchanged test against each replayable historical defect and corrected code. Follow historical verification when old revisions, schemas, configuration, or data are involved.
9. Audit every claimed historical catch card by card. Classify each as:
   • Proven historical code catch: the real pre-fix path or faithful inverse patch fails for the historical reason, and corrected application code passes;
   • Proven historical data catch: the historical malformed state is replayed through the real validator and rejected, while valid historical data passes;
   • Variant required: the framework can catch it only after a named fixture or branch is added;
   • Not covered: different mechanism, insufficient evidence, configuration behaved correctly, or no reproducible defect.
10. Report the denominator, exact proven historical catches, commands, runtime, fixture/cleanup behavior, revisions tested, variants, exclusions, and residual risk. Keep model-mutant results separate from historical catch totals.

Test Design Rules

• Assertions must express externally meaningful invariants, not implementation trivia.
• Prefer a reference model plus an adapter that drives the real application behavior for lifecycle systems. A reference model is an oracle, not evidence of a historical catch by itself.
• Interactive scenario builders must use a bounded, server-approved schema. Do not accept arbitrary commands, test paths, classes, or environment variables from the browser.
• A test that passes only current code is incomplete regression evidence. Exact pre-fix application code or a faithful inverse of its documented fix must make a claimed code-regression test fail.
• When old code cannot run because of schema or runtime drift, use the narrowest faithful alternative: historical schema enclave, inverse patch/mutant, or recorded fixture replay. Label the evidence level.
• If a historical issue lacks a root cause or stable reproduction, do not claim it as covered merely because the test checks a similar symptom or rejects a convenient model mutant.
• Keep code, configuration, and data findings separate. A combined suite may cover all three, but each assertion must identify which boundary failed.

Deliverable

Produce a developer-ready test with:

• one documented CLI or CI command;
• focused unit or integration coverage at the appropriate layer;
• optional explanatory GUI using the same scenario definitions;
• automatic temporary-data cleanup;
• positive and negative control evidence;
• a concise historical coverage matrix with clickable issue links when available.

End with a verdict table containing: total eligible issues reviewed, issues in the selected cluster, historical defects replayed, proven catches, variants not yet proven, exclusions, current-good-code result, and whether the test is a good broad regression. Assertion count and model-mutant count may be reported, but never substituted for proven catches.