Software Regression Testing

software-regression-testing

Summary

Regression testing validates that code changes do not break existing functionality by running the full test suite after modifications. In MS2Query development, this ensures that new features or bug fixes maintain compatibility with the existing MS2Query spectral matching workflow.

When to use

After implementing code changes to MS2Query (e.g., modifying the workflow branching logic for true library matches vs. analog search results), before committing or pushing to the repository. Also required during pull request reviews to verify that new contributions do not degrade existing MS2Query functionality.

When NOT to use

• When you have not yet made any code changes (no changes to test).
• When working on documentation-only updates that do not modify any Python source code.
• Before first understanding what the existing test suite covers; review tests before making changes to ensure your modifications align with expected behavior.

Inputs

• Modified Python source code (MS2Query codebase with feature branch changes)
• Existing unit test suite (located in the repository)

Outputs

• Test execution report (pass/fail status for each test)
• Regression test log (stdout/stderr from test run)
• Confidence that no existing functionality was broken by the changes

How to apply

Execute the full test suite using python setup.py test to verify that all existing tests pass after your code modifications. If any tests fail, inspect the failure output to identify which components were affected by your changes, then revise your implementation to restore passing status. This step should be performed locally before pushing your feature branch and again automatically during continuous integration when the pull request is created. The test results serve as the gate: only push when all tests pass.

Related tools

• Python (Runtime environment for executing regression test suite via setup.py test command)
• pytest or unittest framework (Test execution framework invoked by setup.py to run unit tests)
• GitHub Actions CI (Continuous integration system that automatically runs regression tests on pull requests (CI_build.yml workflow)) — https://github.com/iomega/ms2query/actions/workflows/CI_build.yml
• MS2Query (Target library whose functionality is validated by regression testing) — https://github.com/iomega/ms2query

Examples

python setup.py test

Evaluation signals

• All tests return exit code 0 (successful completion) when running python setup.py test
• No test failures, errors, or skipped tests appear in the test execution output
• The test output confirms that the same number of tests pass before and after your code changes (or increased if you added new tests)
• GitHub Actions CI workflow completes with a green status (all checks pass) after your pull request is created
• Pull request can be merged without test-related blockers on the upstream iomega/ms2query repository

Limitations

• Regression testing only validates existing test coverage; it cannot detect bugs or regressions in untested code paths or new features not covered by current unit tests.
• Test execution time may be lengthy for large codebases; developers should run tests locally before pushing to avoid slow feedback loops in CI.
• Tests may pass on the developer's machine but fail in CI if there are environment-specific dependencies or platform differences (e.g., Windows vs. Linux).
• Regression tests cannot catch performance regressions (e.g., slowdowns) unless explicit performance tests are included in the suite.

Evidence

• [other] make sure the existing tests still work by running python setup.py test: "make sure the existing tests still work by running python setup.py test"
• [other] Run existing tests with python setup.py test to verify no regression.: "Run existing tests with python setup.py test to verify no regression."
• [readme] MS2Query is tested by continous integration on MacOS, Windows and Ubuntu for python version 3.9 and 3.10: "MS2Query is tested by continous integration on MacOS, Windows and Ubuntu for python version 3.9 and 3.10"
• [other] fork the repository to your own Github profile and create your own feature branch off of the latest master commit: "fork the repository to your own Github profile and create your own feature branch off of the latest master commit"
• [other] create the pull request, e.g. following the instructions here: "create the pull request, e.g. following the instructions"