# MCP Test Suite

> Authors test suites for MCP servers using the mcp-test tool: functional YAML assertions, prompt-injection security probes, false-positive tuning, and CI gating. Use when building or maintaining an MCP server that needs more than manual poking in an inspector — especially before exposing it to agent clients you do not control.

- Skill: `zhengqiuyang/mcp-test-suite` (Agent Skill)
- Install (CLI): `npx skillmds@latest add zhengqiuyang/mcp-test-suite`
- Raw SKILL.md: https://api.skillmd.com/api/skills/zhengqiuyang/mcp-test-suite/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- License: MIT
- Author: zhengqiuyang (https://skillmd.com/u/zhengqiuyang)
- Updated: 2026-09-22
- Page: https://skillmd.com/skills/zhengqiuyang/mcp-test-suite

---


# Authoring mcp-test Suites

`mcp-test` runs YAML-defined functional tests and injection probes against a
running MCP server over stdio, with CI-ready exit codes (0 pass, 1 findings,
2 config error). This skill covers the parts the README assumes.

## Start functional, then arm security

1. **Handshake smoke**: `listTools: true` + `toolsContain: [expected-tools]` —
   catches startup config errors instantly.
2. **One call per tool**: empty `assert: {}` asserts "completed" — the cheapest
   possible coverage. Then add `contains`/`jsonSchema`/`maxLatencyMs` for the
   tools that matter.
3. **Security section last**, on the tools that touch untrusted input (files,
   web, tickets, notes).

## Probe philosophy (where FPs come from)

Probes send known injection payloads (instruction-override and credential-file
families — the exact payload table ships in the mcp-test README and probes
source) through tool arguments, then run leak detectors (private-key headers,
token shapes, env-dump shapes, system-prompt markers) on responses. Two design
facts matter when reading results:

- **A tool merely echoing the payload is not a leak** — payloads are stripped
  before detection, including slash-normalized echoes (Windows paths). If you
  see a leak, look for the *detector evidence line*: it names what was found.
- **`suspicious` never fails CI** — it means compliance language + echo, worth
  a look, not a gate.

## FP tuning workflow

A detector firing on your legitimate server:

1. Read the evidence excerpt — which detector, which shape?
2. Drop that probe for the tool (`probes: [ssh-key-exfil, env-exfiltration]`)
   as an immediate unblock, then
3. Fix the underlying imprecision (usually: severity should follow data flow,
   not vocabulary — `env | grep VAR` presence probes are debugging idioms, not
   dumps), add a regression test with the exact false-positive input.

## CI integration

```yaml
- run: npx mcp-test -c mcp-test.yaml --format github
```

`--format github` emits `::error file=...,line=...` annotations pointing at
the failing YAML entry; the exit code gates the build with no extra steps.
Note: `.cmd` server commands on Windows need `command: cmd, args: [/c, ...]`
(spawn is shell-less).

## Schema-drift upgrade gate (companion)

Pin tool versions with `mcp-test snapshot`; before any upgrade run
`mcp-test diff --snapshot old.json --new <candidate>` — it spins both versions
over stdio and classifies breaking (removed tool, +required param, type
change) vs additive. Exit 1 on breaking changes gates the upgrade in CI.