pytest-output-interpretation
Summary
Interpret pytest test execution output to determine whether a Python package's bundled test suite passes or fails, and extract diagnostic information from the test report. This skill is essential for validating package correctness and identifying test failures during development or continuous integration workflows.
When to use
After running pytest tests/ on a Python package (especially one with dev extras installed), you need to determine if the full test suite passed, identify which tests failed, and extract error messages or stack traces for debugging. Use this skill when you have concrete pytest console output or a test report file and need to synthesize a pass/fail verdict and diagnostic summary.
When NOT to use
- The test suite has not yet been run; you need to execute
pytest first.
- You are trying to modify or write new tests rather than interpret existing results.
- The output is from a different test runner (e.g., unittest, nose, tox) — this skill is pytest-specific.
Inputs
- pytest console output (stdout/stderr)
- pytest test report file (e.g., JUnit XML, JSON, or terminal summary)
- test result summary line from pytest
Outputs
- pass/fail verdict (boolean or enum)
- total test count (integer)
- passed test count (integer)
- failed test count (integer)
- skipped test count (integer)
- list of failed test names and error messages (structured)
- diagnostic summary (string)
How to apply
Execute the test suite using pytest tests/ from the package root (after installing in development mode with pip install -e .[dev]), then parse the final summary line and any FAILED markers in the output. Look for the summary statement that reports the total number of passed, failed, skipped, or errored tests (e.g., '42 passed', '2 failed'). If the exit code is 0 and the summary shows no failures, the test suite passed. If failures or errors are reported, read the FAILED section and the traceback for each failing test to extract the assertion error or exception type. Record the specific test names, error messages, and any relevant stack trace context to guide remediation. The test pass/fail status directly validates package correctness and reproducibility.
Related tools
- pytest (test runner; executes the bundled test suite and produces structured output with pass/fail status and failure diagnostics) — https://pytest.readthedocs.io/
- pip (package manager; installs the package in development mode with test dependencies before pytest execution) — https://pip.pypa.io/
- Python (runtime environment; executes pytest and the test suite) — https://www.python.org/
Examples
pytest tests/
Evaluation signals
- Exit code is 0 (success) or non-zero (failure) — the process exit code reliably indicates pass/fail
- The final pytest summary line is present and parseable (e.g., 'X passed', 'X failed', 'X error') — absence suggests incomplete output
- All test node IDs in the FAILED section match patterns in the tests/ directory (tests/test_*.py::TestClass::test_method) — validates test identification
- Error messages and stack traces are present for each failed test and contain recognizable Python exception types (AssertionError, ImportError, etc.) — confirms diagnostic completeness
- Test counts are consistent: passed + failed + skipped ≤ total tests run — validates internal consistency of the report
Limitations
- pytest output format may vary by version, verbosity flag, or plugin configuration (e.g.,
-v, -vv, --tb=short), requiring robust parsing.
- Flaky or nondeterministic tests may pass or fail on successive runs, making a single test run an unreliable validator of package state.
- Pytest may be interrupted or killed before completion, producing truncated output that cannot be fully interpreted.
- Conditional test skips (marked with
@pytest.mark.skip or platform/dependency conditions) mean the absence of a test result does not imply failure.
Evidence
- [other] The biosynfoni repository displays a Tests badge linked to a GitHub Actions workflow (test-biosynfoni.yml) that validates the package's test suite status.: "The biosynfoni repository displays a Tests badge linked to a GitHub Actions workflow (test-biosynfoni.yml) that validates the package's test suite status."
- [other] Run the full test suite using pytest on the tests/ directory (pytest tests/). Verify that all tests pass and report the test outcome.: "Run the full test suite using pytest on the tests/ directory (pytest tests/). Verify that all tests pass and report the test outcome."
- [other] Install the package in editable/development mode using pip with the dev extras (pip install -e .[dev]).: "Install the package in editable/development mode using pip with the dev extras (pip install -e .[dev])."
- [other] You can also run the tests locally with the following command: pytest tests/: "You can also run the tests locally with the following command: pytest tests/"
1---2name: pytest-output-interpretation3description: Use when after running `pytest tests/` on a Python package (especially one with dev extras installed), you need to determine if the full test suite passed, identify which tests failed, and extract error messages or stack traces for debugging.4license: CC-BY-4.05---67# pytest-output-interpretation89## Summary1011Interpret pytest test execution output to determine whether a Python package's bundled test suite passes or fails, and extract diagnostic information from the test report. This skill is essential for validating package correctness and identifying test failures during development or continuous integration workflows.1213## When to use1415After running `pytest tests/` on a Python package (especially one with dev extras installed), you need to determine if the full test suite passed, identify which tests failed, and extract error messages or stack traces for debugging. Use this skill when you have concrete pytest console output or a test report file and need to synthesize a pass/fail verdict and diagnostic summary.1617## When NOT to use1819- The test suite has not yet been run; you need to execute `pytest` first.20- You are trying to modify or write new tests rather than interpret existing results.21- The output is from a different test runner (e.g., unittest, nose, tox) — this skill is pytest-specific.2223## Inputs2425- pytest console output (stdout/stderr)26- pytest test report file (e.g., JUnit XML, JSON, or terminal summary)27- test result summary line from pytest2829## Outputs3031- pass/fail verdict (boolean or enum)32- total test count (integer)33- passed test count (integer)34- failed test count (integer)35- skipped test count (integer)36- list of failed test names and error messages (structured)37- diagnostic summary (string)3839## How to apply4041Execute the test suite using `pytest tests/` from the package root (after installing in development mode with `pip install -e .[dev]`), then parse the final summary line and any FAILED markers in the output. Look for the summary statement that reports the total number of passed, failed, skipped, or errored tests (e.g., '42 passed', '2 failed'). If the exit code is 0 and the summary shows no failures, the test suite passed. If failures or errors are reported, read the FAILED section and the traceback for each failing test to extract the assertion error or exception type. Record the specific test names, error messages, and any relevant stack trace context to guide remediation. The test pass/fail status directly validates package correctness and reproducibility.4243## Related tools4445- **pytest** (test runner; executes the bundled test suite and produces structured output with pass/fail status and failure diagnostics) — https://pytest.readthedocs.io/46- **pip** (package manager; installs the package in development mode with test dependencies before pytest execution) — https://pip.pypa.io/47- **Python** (runtime environment; executes pytest and the test suite) — https://www.python.org/4849## Examples5051```52pytest tests/53```5455## Evaluation signals5657- Exit code is 0 (success) or non-zero (failure) — the process exit code reliably indicates pass/fail58- The final pytest summary line is present and parseable (e.g., 'X passed', 'X failed', 'X error') — absence suggests incomplete output59- All test node IDs in the FAILED section match patterns in the tests/ directory (tests/test_*.py::TestClass::test_method) — validates test identification60- Error messages and stack traces are present for each failed test and contain recognizable Python exception types (AssertionError, ImportError, etc.) — confirms diagnostic completeness61- Test counts are consistent: passed + failed + skipped ≤ total tests run — validates internal consistency of the report6263## Limitations6465- pytest output format may vary by version, verbosity flag, or plugin configuration (e.g., `-v`, `-vv`, `--tb=short`), requiring robust parsing.66- Flaky or nondeterministic tests may pass or fail on successive runs, making a single test run an unreliable validator of package state.67- Pytest may be interrupted or killed before completion, producing truncated output that cannot be fully interpreted.68- Conditional test skips (marked with `@pytest.mark.skip` or platform/dependency conditions) mean the absence of a test result does not imply failure.6970## Evidence7172- [other] The biosynfoni repository displays a Tests badge linked to a GitHub Actions workflow (test-biosynfoni.yml) that validates the package's test suite status.: "The biosynfoni repository displays a Tests badge linked to a GitHub Actions workflow (test-biosynfoni.yml) that validates the package's test suite status."73- [other] Run the full test suite using pytest on the tests/ directory (pytest tests/). Verify that all tests pass and report the test outcome.: "Run the full test suite using pytest on the tests/ directory (pytest tests/). Verify that all tests pass and report the test outcome."74- [other] Install the package in editable/development mode using pip with the dev extras (pip install -e .[dev]).: "Install the package in editable/development mode using pip with the dev extras (pip install -e .[dev])."75- [other] You can also run the tests locally with the following command: pytest tests/: "You can also run the tests locally with the following command: pytest tests/"
HolobiomicsLab