AI Validation

AI Validation & Polish Skill

This skill orchestrates Phase 4 of the mcp-builder pipeline: validating deterministically-generated MCP server code for correctness and suggesting hint-driven improvements. The pipeline transforms an OpenAPI spec into a ToolHive-ready MCP server across four phases — AI Scoping (Phase 1) produces a validated mcp-scope.yaml, Human Review (Phase 2) refines it, Deterministic Code Generation (Phase 3) scaffolds the server, and this skill (Phase 4) validates the generated code against the YAML and spec, then suggests improvements driven by hints from the scoping phase.

Startup

Before beginning the workflow:

1. Determine the base directory for this skill (the absolute path to the directory containing this SKILL.md). The repo root is two levels up: {skill_base_dir}/../... Agent files are at {repo_root}/agents/{agent-name}.md. Assets are at {skill_base_dir}/assets/. Do not share these internal paths with the user.

2. All uv and task commands must run from {repo_root} (where pyproject.toml lives).

3. Clone or locate the reference repos that agents need for grounding their checks:

   • ToolHive (stacklok/toolhive) — CRD schemas, auth patterns. Check if a local clone exists nearby (e.g., sibling directory). If not, clone to a temp directory:

     gh repo clone stacklok/toolhive /tmp/toolhive-ref -- --depth 1
     

   • mcp-template-py (stacklok/mcp-template-py) — the base Python MCP server template. Same approach:

     gh repo clone stacklok/mcp-template-py /tmp/mcp-template-py-ref -- --depth 1
     

   Record the absolute paths to both repos. These will be passed to agents.

Workflow

Given a generated project directory, mcp-scope.yaml, and OpenAPI spec path ($ARGUMENTS), execute the following steps:

1. Collect and verify inputs
2. Code validation (agent)
3. Validation gate (user gate)
4. Polish suggestions (agent)
5. Polish application gate (user gate)
6. Final build verification (if code was modified)
7. Present results

Step 1: Collect and Verify Inputs

1. Parse $ARGUMENTS for three required paths:

   • Generated project directory — output from Phase 3 code generation
   • mcp-scope.yaml — the scope file that drove generation
   • OpenAPI spec — the original spec file

   If fewer than 3 arguments are provided, ask the user for the missing paths.

2. Verify all three paths exist. If any is missing, tell the user and exit.

3. Read the server.name from mcp-scope.yaml. Derive the module name: replace hyphens with underscores, append _mcp (e.g., google-drive → google_drive_mcp).

4. Verify the generated project has the expected structure by checking these files exist:

   • src/{module_name}/api/tools.py
   • src/{module_name}/api/mcp_builder.py
   • src/{module_name}/client.py
   • deploy/mcpserver.yaml
   • pyproject.toml
   • Dockerfile

   If key files are missing, note which ones and continue — the validator will report them as failures.

5. Create a working directory for output files: {cwd}/validation-output-{date}/

Step 2: Code Validation (Agent)

Spawn a code-validator sub-agent using the Agent tool. Pass file paths — do NOT read or paste file contents into the prompt. The agent reads everything itself.

Agent tool parameters:
- subagent_type: [path to code-validator agent: {repo_root}/agents/code-validator.md]
- description: "Validate generated MCP server code"
- prompt: |
    Validate the generated MCP server code for correctness.

    CONTEXT:
    Pipeline context path: [absolute path to {skill_base_dir}/../ai-scoping/assets/pipeline-context.md]
    Report template path: [absolute path to {skill_base_dir}/assets/validation-report-template.md]
    Working directory: [absolute path to validation-output/]

    SERVER METADATA:
    Server name: [from YAML server.name]
    Module name: [derived module name]

    FILE PATHS:
    Project directory: [absolute path to generated project]
    MCP scope YAML: [absolute path to mcp-scope.yaml]
    OpenAPI spec: [absolute path to spec file]

    REFERENCE REPOS:
    ToolHive repo: [absolute path to toolhive clone]
    mcp-template-py repo: [absolute path to mcp-template-py clone]
- mode: acceptEdits
- run_in_background: false

The code-validator agent will read all files, run the Docker build check, and write {working_dir}/validation-report.md.

Step 3: Validation Gate

1. Read {working_dir}/validation-report.md — pay particular attention to the Build Verification row and any error-severity failures.

2. Present the validation summary to the user using the structure below.

   ## Validation Summary
   
   **Report:** {path to validation-report.md}
   
   - Total checks: {X} passed, {Y} failed, {Z} skipped
   
   ### 🏗️ Docker build: {✅ PASS / ❌ FAIL / ⏭️ SKIP}
   
   <copy reason build failed or skipped>
   

   If there are error-severity failures from other checks, list each one prominently under a separate ### ❌ Errors heading with the check ID, file, and one-line fix description from Detailed Findings.

3. Run the appropriate user gate based on what the report contains:

   Case A — error-severity failures exist (USER GATE — wait for response):

   Use AskUserQuestion. Explain how many errors were found and — if the build also failed or was skipped — mention that. Ask how the user wants to proceed. Offer these paths:

   • Have AI fix the errors (spawn the fix agent, then re-validate)
   • Proceed to polish despite the errors
   • Fix manually and re-run the skill later

   Do NOT proceed past this step until the user responds.

   Case B — no error-severity failures, but build is FAIL or SKIP (USER GATE — wait for response):

   Use AskUserQuestion. Explain the situation (build failed or was skipped, include the reason from the Build Verification row) and make clear this is not a blocker for the rest of the flow. Ask what the user wants to do. Offer these paths:

   • Retry the build (re-run the code-validator agent, but instruct it to re-run only the build check and reuse the existing report for everything else)
   • Have AI investigate and fix the build (only when the build actually FAILED — spawn the fix agent scoped to the build, then re-validate)
   • Acknowledge and continue to polish
   • Stop here

   Do NOT proceed past this step until the user responds.

   Case C — no errors and build is PASS:

   Tell the user all checks passed (including build) and automatically proceed to Step 4 (polish suggestions). No gate needed.

Step 3b: AI Error Fixing (conditional)

If the user chose "Have AI fix the errors":

1. Spawn a sub-agent to fix the errors. Pass file paths — do NOT paste file contents.

Agent tool parameters:
- description: "Fix validation errors in generated code"
- prompt: |
    Fix the validation errors in the generated MCP server project.

    PROJECT DIRECTORY: [absolute path]
    MODULE NAME: [module name]

    Read the validation report at: [absolute path to {working_dir}/validation-report.md]
    Read the mcp-scope.yaml at: [absolute path to mcp-scope.yaml]

    For each FAIL item in the Detailed Findings section:
    1. Read the file mentioned in the finding
    2. Apply the fix described
    3. Verify the fix is consistent with the YAML and the rest of the codebase

    Only fix items marked FAIL. Do not make other changes.
- mode: acceptEdits
- run_in_background: false

2. After the agent completes, re-run validation by looping back to Step 2 (re-spawn the code-validator agent).

3. Present the updated results. If errors remain, ask the user again (same choices). Do not loop more than 2 fix attempts — if errors persist after 2 rounds, ask the user to fix manually.

Step 4: Polish Suggestions (Agent)

Spawn a polish-suggester sub-agent using the Agent tool. Pass file paths — do NOT read or paste file contents into the prompt.

Agent tool parameters:
- subagent_type: [path to polish-suggester agent: {repo_root}/agents/polish-suggester.md]
- description: "Suggest hint-driven code improvements"
- prompt: |
    Analyze the generated MCP server code and suggest improvements based on YAML hints.

    CONTEXT:
    Pipeline context path: [absolute path to {skill_base_dir}/../ai-scoping/assets/pipeline-context.md]
    Working directory: [absolute path to validation-output/]
    Validation report path: [absolute path to {working_dir}/validation-report.md]

    SERVER METADATA:
    Server name: [from YAML]
    Module name: [derived module name]

    FILE PATHS:
    Project directory: [absolute path to generated project]
    MCP scope YAML: [absolute path to mcp-scope.yaml]
    OpenAPI spec: [absolute path to spec file]

    REFERENCE REPOS:
    ToolHive repo: [absolute path to toolhive clone]
    mcp-template-py repo: [absolute path to mcp-template-py clone]
- mode: acceptEdits
- run_in_background: false

The polish-suggester agent will read all files and write {working_dir}/polish-suggestions.md.

Step 5: Polish Application Gate (USER GATE)

1. Read {working_dir}/polish-suggestions.md.

2. Present the suggestions to the user using the format below. Do not flatten the report into one-line bullets — the user needs to be able to judge severity and decide fix-or-skip from the chat output alone.

   2a. Overview block — a severity-count table plus an at-a-glance index table:

   ## Polish Suggestions ({N} total)
   
   **Report:** {path to polish-suggestions.md}
   
   | Severity | Count | What it means |
   |----------|-------|---------------|
   | 🟠 high     | {N} | Tool is broken OR LLM callers are very likely to misuse it |
   | 🟡 medium   | {N} | Robustness / quality improvement |
   | ⚪ low      | {N} | Minor polish |
   
   | #  | Severity | Category | Tool(s) | Summary |
   |----|----------|----------|---------|---------|
   | P1 | 🟠 high   | ... | ... | ... |
   | P2 | 🟡 medium | ... | ... | ... |
   | …  | …        | …        | …       | …       |
   

   2b. Per-suggestion detail — for EACH suggestion in the report, render:

   ### P{n}: {title}  —  {severity emoji} {severity}
   
   | Field | Value |
   |-------|-------|
   | Category | {category} |
   | Affected | `{tool(s)}` |
   | File(s)  | `{paths}` |
   | Hint     | {hint text or "AI-analyzed"} |
   
   **Problem:** {copy the Problem paragraph from the report verbatim}
   
   **Impact if unfixed:** {copy verbatim}
   
   **Proposed fix:** {copy verbatim}
   
   <details><summary>View diff</summary>
   
   ```python
   # Before
   {before snippet}
   

   # After
   {after snippet}
   

   Required fields per suggestion: severity, category, affected tool(s), file(s), problem, impact, proposed fix. If the report is missing any of these fields, say so explicitly to the user instead of silently omitting — it means the polish-suggester agent produced an incomplete suggestion and should be re-run.

   Ordering: render high-severity suggestions first, then medium, then low. Within a severity tier, preserve the P-number order from the report. When a high-severity suggestion flags a functional bug in its Problem paragraph, mention that fact when you introduce it so the user understands it isn't just a docstring nit.

   Length: do not truncate Problem / Impact / Proposed fix — they are the whole point. DO wrap long diffs in <details> so the chat stays scannable.

   If there are no suggestions, say so briefly and skip to Step 7 (Step 6 is a no-op when no code was modified).

3. Use AskUserQuestion to ask the user how they want to handle the suggestions. Summarize the counts by severity, then offer these paths:

   • Have AI apply all suggestions
   • Have AI apply selected suggestions (user names the P-numbers)
   • Review manually
   • Skip

   If any high-severity suggestions flagged themselves as functional bugs in the Problem paragraph, mention that in the question so the user knows skipping will leave those behind. Otherwise keep the framing neutral.

Do NOT proceed past this step until the user responds.

Step 5b: AI Polish Application (conditional)

If the user chose to have AI apply suggestions (all or selected):

1. If "selected", ask the user which suggestion IDs to apply (e.g., "P1, P3, P5").

2. Spawn a sub-agent to apply the suggestions. Pass file paths — do NOT paste suggestion contents.

Agent tool parameters:
- description: "Apply polish suggestions to generated code"
- prompt: |
    Apply polish suggestions to the generated MCP server project.

    PROJECT DIRECTORY: [absolute path]
    MODULE NAME: [module name]

    Read the suggestions at: [absolute path to {working_dir}/polish-suggestions.md]
    [If selected: "Only apply suggestions: P1, P3, P5"]

    For each suggestion to apply:
    1. Read the target file listed in the suggestion
    2. Find the "Before" code pattern
    3. Replace it with the "After" code
    4. Verify the change is syntactically valid

    Only apply the listed suggestions. Do not make other changes.
- mode: acceptEdits
- run_in_background: false

Step 6: Final Build Verification (conditional)

Skip this step entirely if no code was modified — no fixes were applied in Step 3b AND no polish was applied in Step 5b. The validation report's existing Build Verification result is still accurate and stands as-is.

Otherwise, rebuild the Docker image to confirm the post-fix / post-polish code still builds. The validator ran its build check against the generator's original output; edits applied after that (either error fixes or polish) may have introduced problems the initial build can't catch.

1. Run the build from the generated project directory:

   cd {project_dir} && docker build -t {server_name}-mcp:validation-test . 2>&1
   

2. Append a Final Build Verification subsection to {working_dir}/validation-report.md (do not overwrite the original Build Verification row — the reader should be able to see both the pre-fix and post-fix results). Record PASS/FAIL/SKIP and, for FAIL or SKIP, the concrete reason including enough build output for the user to understand what broke.

3. If the final build failed, present the failure to the user with AskUserQuestion. Summarize what code was modified (errors fixed, polish applied, or both) and paste the failing build excerpt. Offer:

   • Have AI investigate and fix the build (spawn an inline fix agent scoped to the build failure, then re-run this step)
   • Fix manually and re-run the skill later
   • Acknowledge and continue to Step 7

   Do not loop more than 2 AI-fix attempts here. If the build still fails after 2 rounds, ask the user to fix manually.

4. If the final build passed or was skipped (Docker not installed etc.), continue to Step 7 without a gate.

Step 7: Present Results

Present the user with:

• Path to {working_dir}/validation-report.md
• Path to {working_dir}/polish-suggestions.md (if generated)
• Summary:
  • Validation: X checks passed, Y failed
  • Build: PASS/FAIL/SKIP (report the final build outcome from Step 6 if it ran; otherwise the validator's original result)
  • Polish: N suggestions generated, M applied (if any)
  • Files modified (if any fixes or polish were applied)

Error Handling

Important Notes

Sub-Agent Invocation

• Use the Agent tool with the appropriate subagent_type for code-validator and polish-suggester
• Fix/apply agents use inline prompts (no dedicated agent file) since they're straightforward edit tasks
• Always set run_in_background: false and mode: acceptEdits
• Pass file paths in the prompt — agents read files themselves. Never paste file contents into agent prompts.

Working Directory

• All output files go in {cwd}/validation-output-{date}/
• This includes validation-report.md and polish-suggestions.md

Read-Only by Default

• The skill only modifies the generated project if the user explicitly chooses "Have AI fix" or "Have AI apply"
• Validation and suggestion generation are entirely read-only operations