# Edt MCP E2e Testing

> How to write/run the AUTOMATED black-box e2e suite (tests/e2e/) that covers every EDT-MCP tool (62 today) against a live server with git-fixture isolation, happy + negative + error-quality coverage, and an anti-cheat bar. Use when adding/editing a test in tests/e2e/, adding a new tool (the coverage ratchet requires a test), or running the suite. (For the older MANUAL per-tool reference checklists see edt-mcp-testing; for Java unit/build see edt-mcp-build-test.)

- Skill: `ditrixnew/edt-mcp-e2e-testing-2` (Agent Skill)
- Install (CLI): `npx skillmds@latest add ditrixnew/edt-mcp-e2e-testing-2`
- Raw SKILL.md: https://api.skillmd.com/api/skills/ditrixnew/edt-mcp-e2e-testing-2/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: ditrixnew (https://skillmd.com/u/ditrixnew)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/ditrixnew/edt-mcp-e2e-testing-2

---


# EDT-MCP — automated black-box e2e suite (tests/e2e/)

> **The full, authoritative, always-current guide is `tests/e2e/SKILL.md` in the repo. READ IT before writing or running a test** — it has the harness API, the worked examples, and the anti-cheat anti-pattern list. This skill is the entrypoint + the rules in brief; the repo file wins on any disagreement (it lives with the harness).

## What this is
A Python (stdlib-only) **black-box** suite: a real MCP client hits the **live** server on `:8765` and asserts the **real** effect. One file per tool: `tests/e2e/tools/test_<tool>.py`. Shared base: `harness.py`. Runner: `run_all.py` (serial, `--junit-xml`, per-test timeout). Every registered tool covered (62 today; the coverage ratchet `test_coverage_ratchet.py` fails the suite if a `tools/list` tool has no test). Different layer from Java/JUnit (which is headless and can't touch a real project).

## Run it
```
python tests/e2e/run_all.py --project TestConfiguration --junit-xml tests/e2e/e2e-results.xml
```
Needs a LIVE server on `:8765` + `TestConfiguration` loaded (the dev EDT copy — see the project memory / edt-redeploy loop). Execution is **serial** (one fixture, one git tree); authoring can be parallel (one agent per tool file). `--filter <substr>` runs a subset.

## The rules in brief (details in tests/e2e/SKILL.md)
1. **One file per tool**, `tests/e2e/tools/test_<tool>.py`, importing ONLY from `harness`. Adding a new MCP tool? add its test file — the **coverage ratchet** fails the suite otherwise.
2. **git-fixture isolation:** `TestConfiguration` is a committed fixture. The orchestrator resets it before every test (`git reset` + `git checkout HEAD --` + `git clean -fd`). Read tools must leave it clean (`assert_no_diff()`).
3. **Verify the effect the RIGHT way per `kind=`:**
   - `read` → `assert_no_diff()` + content.
   - `write` (file-write, write_module_source) → `assert_diff_contains()` (on disk).
   - `write-metadata` (create/add/delete/rename) → **MODEL READ-BACK + on-disk structure.** Call a read tool and assert the change in the model, AND assert it landed on disk via `poll_diff_contains` of the **structural element** (`<name>X</name>`, `<catalogs>Type.X</catalogs>` — not the bare name). Metadata writes now persist to disk via forceExport (~1-2s lag → poll). The orchestrator runs `reset_fixture`+`reset_model` after these.
   - `action` → usually `assert_no_diff()` + status; never run a destructive happy mutation that corrupts the fixture.
   - **env-dependent** (form-render JVM flag, LanguageTool, debug no-session) → env-robust branch: real result OR a clear actionable **sentinel**.
4. **Negative coverage is MANDATORY** (not just happy): invalid path/object, every XOR/conditional/enum, missing-required, boundaries. Assert **error QUALITY** (`assert_error_quality`): names the bad value + actionable next step. A vague/non-actionable tool error → `# AUDIT:` note + a fix-card; **do NOT weaken the test** to a bad error.
5. **Anti-cheat (mutation thinking):** "would this test FAIL if the tool were broken?" If not, it's a CHEAT. No trivial asserts, no `assert_no_diff()`-only, no swallowed exceptions, no tolerant matching. A separate anti-cheat reviewer rules REAL/CHEAT.

## Don't
- Don't add pip dependencies (stdlib only; the client is kept spec-conformant by hand on purpose).
- Don't assert a metadata-write effect with the BARE name in the diff (it false-matches the parent's collection reference) — assert the structural element via `poll_diff_contains`, paired with a model read-back. (Metadata writes DO persist now, with a ~1-2s lag.)
- Don't parallelize the RUN (serial only). Don't call `reset_fixture`/`reset_model` in a test (the orchestrator owns them).

## Protocol conformance (a separate gate)
This suite tests tool **business logic**. The MCP **protocol** itself (initialize handshake, capabilities/version negotiation, `Mcp-Session-Id`, `Accept`/`Content-Type`, `isError`, `ping`, SSE) is a different layer, checked by the official `modelcontextprotocol/conformance` suite — see `tests/conformance/` (`tests/conformance/README.md`). Run: `npx @modelcontextprotocol/conformance@latest server --url http://127.0.0.1:8765/mcp --spec-version 2025-11-25 --expected-failures tests/conformance/baseline.yml` → green when only the pinned (intentional) gaps fail. Don't mix the two gates.

Related: `edt-mcp-tool-conventions` (param/error/output canon), `edt-mcp-bilingual` (ru/en), `edt-mcp-build-test` (Java unit/build), `edt-mcp-testing` (manual per-tool checklists), `edt-mcp-ready-to-deploy` (final pre-merge checklist).