MCP Tool Validator

MCP Tool Validator

This skill provides a complete, drop-in test harness for validating MCP server tools after code changes. It enforces a strict feedback loop: deploy → validate every tool → fix failures → redeploy → re-validate, capped at 5 iterations, before a PR is raised.

Works for both single-repo and monorepo layouts. Override the defaults table below to match your project structure.

Configuration defaults (override per project)

Monorepo note: If your MCP server lives in a subdirectory (e.g. mcp/<server-name>/), prefix all paths above with that subdirectory. Example: SCENARIOS_FILE=mcp/my-server/tests/scenarios.yaml.

When to use this skill

Use this skill when ANY of the following are true:

• Code changes were made to your MCP server src/ and the server has been rebuilt/redeployed
• Health check passes but tool-level behavior is not yet verified
• The user wants autonomous fix-and-retry behavior before raising a PR touching MCP code

Do NOT use this skill for:

• Pure unit testing of internal handler logic (use npm test or pytest in your MCP server directory)
• Non-MCP HTTP APIs (use standard contract testing)
• Load/performance testing (out of scope)

Core principle

A green health check does not mean the tools work. This skill closes that gap by spawning a real MCP client, calling every tool with declared scenarios, and asserting on protocol shape AND semantic content.

Quick start (the 5-step procedure)

1. Inventory tools — Run python scripts/validate_coverage.py --server-url $SERVER_URL --scenarios tests/scenarios.yaml --transport http from inside your MCP server directory to verify coverage.
2. Author scenarios — For each tool, you need: 1 happy path, 1 invalid input, 1 edge case. See references/scenario_schema.md to author or extend scenarios.
3. Drop in the test client — tests/mcp_test_client.py is the client. No modifications needed for standard use.
4. Wire the loop — Use tests/run_validation.sh as the orchestrator. It handles build → deploy → wait → validate → report.
5. Enforce the contract — Copy the feedback-loop block from references/feedback_loop.md into your project's CLAUDE.md.

Lazy-loaded references

Read these only when needed for the current task:

• references/client_skeleton.md — Read when implementing or modifying the test client itself (transport details, session lifecycle, stdio vs HTTP)
• references/scenario_schema.md — Read when authoring scenarios or extending the assertion DSL
• references/assertion_library.md — Read when a scenario needs an assertion type not in the starter set
• references/feedback_loop.md — Read when configuring the autonomous fix-redeploy-retest loop in CLAUDE.md
• references/docker_orchestration.md — Read when health-check waits, port conflicts, or compose-file orchestration cause issues

Coverage rules (non-negotiable)

Every tool registered via tools/list MUST have:

• At least 1 happy-path scenario (name contains happy_path)
• At least 1 input-validation/error scenario (name contains invalid_input)
• At least 1 edge case (any other name)

Run python scripts/validate_coverage.py --server-url $SERVER_URL --scenarios tests/scenarios.yaml before committing. It fails if any tool is missing scenarios.

Failure-mode guards

These guards prevent the most common ways autonomous loops produce false-green results:

1. No silent xfail — Marking a scenario as expected-fail requires a # REASON: <one line> comment AND user approval in the conversation.
2. No scenario softening — Edits to scenarios.yaml require commit message tag [scenario-update] and a one-line justification.
3. Loop cap — Default 5 fix-redeploy iterations. After cap, STOP and report with file:line evidence. Do NOT raise PR.
4. Determinism check — Each scenario runs twice on first failure. If results differ, the scenario is flaky → fix the scenario or the tool, not the loop.
5. Coverage gate — PR creation is blocked if validate_coverage.py exits non-zero.

Output contract

After running validation, produce a report with:

• Total tools discovered, total scenarios run, pass/fail counts
• For each failure: tool name, scenario name, expected vs actual, suspected root cause file:line
• Iteration count if loop was used
• Final verdict: GREEN (raise PR) | RED (stop, report) | FLAKY (investigate)