Test Review

Katalon Test Review

Use this skill for the review stage: inspect coverage, quality, and reliability so weak tests do not enter the pipeline. The output is a verdict with named weak cases, never a metric dump.

Availability Boundary

• Available via MCP: coverage review (fetch_requirement_data, find_test_cases_by_requirement, fetch_test_configuration_data), quality review (fetch_test_case_data), reliability review (fetch_test_stability_data, find_test_results), and environment readiness (read_auts).
• Not directly available: code/object review, local debug, StudioAssist Ask — these are Studio-desktop operations. For code-lane review, defer to test-case-to-playwright / playwright-execute. Use Browser/Playwright only for AUT sanity checks when asked.

Review Workflow

+---------------------+     +----------------------+     +----------------------+
| Coverage review     | --> | Quality review       | --> | Reliability review   |
| reqs + config       |     | case design signals  |     | flakiness/stability  |
+---------------------+     +----------------------+     +----------------------+
                                                                   |
                                                                   v
                                                         +----------------------+
                                                         | Verdict + weak cases |
                                                         +----------------------+

Steps and tool rules

1. Coverage review. fetch_requirement_data + find_test_cases_by_requirement for requirement coverage; fetch_test_configuration_data for browser/platform coverage. Flag orphan requirements and under-covered configurations.
2. Quality review. fetch_test_case_data for design signals; read representative cases with read_test_case when a signal is ambiguous. Flag non-atomic cases (many assertions), missing negative/boundary variants, and vague expected results.
3. Reliability review. fetch_test_stability_data for flakiness; find_test_results for recent pass/fail history. Flag probabilistically flaky cases that will erode pipeline trust.
4. Environment readiness. read_auts to confirm an executable AUT exists for the suite.
5. Verdict. One of Approve / Approve with fixes / Reject for pipeline, followed by the specific cases to fix and why. State the risk if approving with known gaps.

Verdict rubric

• Approve: coverage meets the plan, no flaky cases in the critical path, expected results are observable.
• Approve with fixes: ship-able but list the exact cases needing a fix (flaky, non-atomic, weak expected result) and the owner action.
• Reject for pipeline: orphan critical requirements, or flaky cases in the smoke/regression core — fix before the suite runs.

Prompt recipes

• Review the regression suite for release 3.2: is it ready for the pipeline? Give a verdict and list weak cases.
• Check requirement and configuration coverage for project X and flag anything under-covered.
• Which cases in the smoke suite are flaky enough to reject before we wire them into CI?

Hand-offs

• Fixes needed -> create-test-cases (redesign) or test-maintenance (repair/flaky).
• Approved -> execute-test.
• Ship decision after execution -> release-analyze.

Read references/review-rubric.md before issuing a verdict. Consult the orchestrator's references/unavailable-capabilities.md for boundaries.