Golden Test Reviewer

Golden Test Reviewer

Skill for reviewing and evaluating gqlkit golden test cases using parallel subagent evaluation.

Purpose

Systematically evaluate golden test cases under packages/cli/src/gen-orchestrator/testdata/, identify issues, and provide concrete fix suggestions. Each test case is evaluated by a dedicated subagent in parallel for efficiency.

Workflow

Step 1: Select Review Target

When the review target is not explicitly specified, ask the user (using the runtime's available interaction mechanism) to choose from the following options:

1. All: Review all test cases
2. Category: Review test cases in a specific category only
   • Basic types (basic-*)
   • Scalars (scalar-*, branded-scalar)
   • Directives (directive-*, deprecated-*)
   • Interfaces (interface-*)
   • Unions (union-*)
   • Input types (input-*, oneof-*)
   • Default values (default-value-*)
   • Errors (*-errors, type-error-*)
3. Diff: Review only test cases with changes between current branch and default branch (main)
4. PR/Branch Review: Review golden tests affected by a specific PR or branch
   • Reviews test cases modified in the PR/branch
   • Evaluates test coverage for features changed in the PR/branch

Detecting Review Mode

When triggered by code review requests (e.g., "review PR #123", "review this branch"), automatically use PR/Branch Review mode.

• If PR number is provided: Use gh pr diff <number> to get the diff
• If branch name is provided or current branch differs from main: Use git diff main...<branch>

Getting Changed Test Cases

# Get test cases with changes (for Diff mode)
git diff main --name-only -- packages/cli/src/gen-orchestrator/testdata/ | \
  sed 's|packages/cli/src/gen-orchestrator/testdata/||' | \
  cut -d'/' -f1 | sort -u

# Get test cases with changes from PR (for PR/Branch Review mode)
gh pr diff <PR_NUMBER> --name-only | \
  grep '^packages/cli/src/gen-orchestrator/testdata/' | \
  sed 's|packages/cli/src/gen-orchestrator/testdata/||' | \
  cut -d'/' -f1 | sort -u

# Get test cases with changes from branch (for PR/Branch Review mode)
git diff main...<branch> --name-only -- packages/cli/src/gen-orchestrator/testdata/ | \
  sed 's|packages/cli/src/gen-orchestrator/testdata/||' | \
  cut -d'/' -f1 | sort -u

Step 1.5: PR/Branch Review - Analyze Diff and Identify Affected Features (PR/Branch Review mode only)

When in PR/Branch Review mode, perform additional analysis to identify affected features and evaluate test coverage.

1.5.1: Get Full Diff

# For PR
gh pr diff <PR_NUMBER>

# For branch
git diff main...<branch>

1.5.2: Identify Affected CLI Components

Analyze the diff to identify which CLI components are affected. Map changed files to feature categories:

1.5.3: Map Features to Test Categories

Based on affected components, determine which test categories should have coverage:

1.5.4: Check Test Coverage

Compare the expected test categories with actually modified/added test cases. Flag gaps where:

• A feature is modified but no corresponding test case is added/modified
• A new feature is added but no test case covers it

This analysis will be included in the final report (Step 3).

Step 2: Launch Subagents for Parallel Evaluation

For each test case to be reviewed, create a separate subtask with an isolated context (subagent/subtask model) using the runtime's available execution mechanism.

IMPORTANT:

• Treat each test case as an independently designed subtask with clear inputs/outputs.
• Execute independent subtasks in parallel when supported by the runtime.
• Pass only minimal required context to each subtask (target test case, relevant diff, and evaluation criteria) to keep context isolated.
• Normalize and aggregate subtask outputs in the final report.

Subagent Prompt

Read the prompt template from references/subagent-prompt.md and use it for each subagent.

For standard review (All, Category, Diff modes):

• Use the first template in references/subagent-prompt.md
• Replace {test-case-name} with the actual test case name
• Evaluates against 6 criteria (naming, source quality, schema accuracy, resolvers, diagnostics, MECE)

For PR/Branch Review mode:

• Use the second template (PR/Branch Review Mode) in references/subagent-prompt.md
• Replace placeholders:
  • {test-case-name} - actual test case name
  • {pr-number} or {branch-name} - PR number or branch name
  • {diff-summary} - summary of changed files in the PR/branch
  • {test-case-diff} - diff content for the specific test case
• Evaluates against 7 criteria (standard 6 + change appropriateness)

Example: Launching Multiple Analysis Tasks (Standard Mode)

When reviewing test cases interface-basic, union-type, and field-resolver:

Use the runtime's parallel task mechanism to evaluate these test cases as isolated subtasks concurrently:

• interface-basic
• union-type
• field-resolver

Example: Launching Analysis Tasks (PR/Branch Review Mode)

When reviewing test cases modified in PR #123:

Create one isolated subtask per modified test case (1 test case = 1 isolated subtask).

For each subtask, provide only:

• target test case name
• PR number or branch name
• diff summary limited to files relevant to that test case
• per-test-case diff
• required evaluation criteria

Do not share full PR/branch-wide cross-case context with every subtask. Cross-case reasoning (e.g., coverage gaps across categories) must be performed only in Step 3 aggregation.

Step 3: Aggregate Results and Generate Report

After all subagents complete:

1. Parse JSON results from each subagent
2. Generate summary table showing all test cases at a glance:

## Review Summary

| Test Case | Naming | Source | Types | Conversion | Resolvers | Diagnostics | MECE | Issues |
|-----------|--------|--------|-------|------------|-----------|-------------|------|--------|
| interface-basic | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ⚠️ | 1 |
| union-type | ✅ | ⚠️ | ✅ | ✅ | ✅ | ✅ | ✅ | 1 |
| field-resolver | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | 0 |

3. List issues by severity (errors first, then warnings):

## Issues Found

### Errors
- **test-case-name**: Description of error (Suggestion: ...)

### Warnings
- **test-case-name**: Description of warning (Suggestion: ...)

4. Provide detailed reports for test cases with issues (expand from JSON)

   Note: Perform repository-wide or cross-case synthesis only at aggregation time, not inside isolated subtasks.

5. (PR/Branch Review mode only) Include Test Coverage Analysis:

## Test Coverage Analysis

### Affected Components
| Component | Changed Files |
|-----------|---------------|
| type-extractor | `interface.ts`, `union.ts` |
| schema-generator | `builder.ts` |

### Expected vs Actual Test Coverage
| Feature Category | Expected | Actual | Status |
|------------------|----------|--------|--------|
| Interface handling | `interface-*` | `interface-basic` (modified) | ✅ Covered |
| Union handling | `union-*` | (none) | ⚠️ Missing |

### Coverage Gaps
- **Union handling**: Changes to `type-extractor/union.ts` but no union test cases modified or added
  - Suggestion: Consider adding/updating `union-*` test cases to cover the changes

### Coverage Summary
- Components changed: 2
- Test categories expected: 2
- Test categories covered: 1
- Coverage gaps: 1

Evaluation Ratings

• ✅ Good ("good"): No issues
• ⚠️ Warning ("warning"): Room for improvement (functionally correct)
• ❌ Error ("error"): Fix required

Test Case Categories

Additional Resources

Reference Files

• references/subagent-prompt.md - Prompt template for subagent evaluation
• references/evaluation-criteria.md - Detailed criteria and judgment examples
• references/testcase-structure.md - Test case structure and conventions

Converted and distributed by TomeVault — claim your Tome and manage your conversions.