EDT-MCP — automated black-box e2e suite (tests/e2e/)
The full, authoritative, always-current guide is tests/e2e/SKILL.md in the repo. READ IT before writing or running a test — it has the harness API, the worked examples, and the anti-cheat anti-pattern list. This skill is the entrypoint + the rules in brief; the repo file wins on any disagreement (it lives with the harness).
What this is
A Python (stdlib-only) black-box suite: a real MCP client hits the live server on :8765 and asserts the real effect. One file per tool: tests/e2e/tools/test_<tool>.py. Shared base: harness.py. Runner: run_all.py (serial, --junit-xml, per-test timeout). Every registered tool covered (62 today; the coverage ratchet test_coverage_ratchet.py fails the suite if a tools/list tool has no test). Different layer from Java/JUnit (which is headless and can't touch a real project).
Run it
python tests/e2e/run_all.py --project TestConfiguration --junit-xml tests/e2e/e2e-results.xml
Needs a LIVE server on :8765 + TestConfiguration loaded (the dev EDT copy — see the project memory / edt-redeploy loop). Execution is serial (one fixture, one git tree); authoring can be parallel (one agent per tool file). --filter <substr> runs a subset.
The rules in brief (details in tests/e2e/SKILL.md)
- One file per tool,
tests/e2e/tools/test_<tool>.py, importing ONLY from harness. Adding a new MCP tool? add its test file — the coverage ratchet fails the suite otherwise.
- git-fixture isolation:
TestConfiguration is a committed fixture. The orchestrator resets it before every test (git reset + git checkout HEAD -- + git clean -fd). Read tools must leave it clean (assert_no_diff()).
- Verify the effect the RIGHT way per
kind=:
read → assert_no_diff() + content.write (file-write, write_module_source) → assert_diff_contains() (on disk).write-metadata (create/add/delete/rename) → MODEL READ-BACK + on-disk structure. Call a read tool and assert the change in the model, AND assert it landed on disk via poll_diff_contains of the structural element (<name>X</name>, <catalogs>Type.X</catalogs> — not the bare name). Metadata writes now persist to disk via forceExport (~1-2s lag → poll). The orchestrator runs reset_fixture+reset_model after these.action → usually assert_no_diff() + status; never run a destructive happy mutation that corrupts the fixture.- env-dependent (form-render JVM flag, LanguageTool, debug no-session) → env-robust branch: real result OR a clear actionable sentinel.
- Negative coverage is MANDATORY (not just happy): invalid path/object, every XOR/conditional/enum, missing-required, boundaries. Assert error QUALITY (
assert_error_quality): names the bad value + actionable next step. A vague/non-actionable tool error → # AUDIT: note + a fix-card; do NOT weaken the test to a bad error.
- Anti-cheat (mutation thinking): "would this test FAIL if the tool were broken?" If not, it's a CHEAT. No trivial asserts, no
assert_no_diff()-only, no swallowed exceptions, no tolerant matching. A separate anti-cheat reviewer rules REAL/CHEAT.
Don't
- Don't add pip dependencies (stdlib only; the client is kept spec-conformant by hand on purpose).
- Don't assert a metadata-write effect with the BARE name in the diff (it false-matches the parent's collection reference) — assert the structural element via
poll_diff_contains, paired with a model read-back. (Metadata writes DO persist now, with a ~1-2s lag.)
- Don't parallelize the RUN (serial only). Don't call
reset_fixture/reset_model in a test (the orchestrator owns them).
Protocol conformance (a separate gate)
This suite tests tool business logic. The MCP protocol itself (initialize handshake, capabilities/version negotiation, Mcp-Session-Id, Accept/Content-Type, isError, ping, SSE) is a different layer, checked by the official modelcontextprotocol/conformance suite — see tests/conformance/ (tests/conformance/README.md). Run: npx @modelcontextprotocol/conformance@latest server --url http://127.0.0.1:8765/mcp --spec-version 2025-11-25 --expected-failures tests/conformance/baseline.yml → green when only the pinned (intentional) gaps fail. Don't mix the two gates.
Related: edt-mcp-tool-conventions (param/error/output canon), edt-mcp-bilingual (ru/en), edt-mcp-build-test (Java unit/build), edt-mcp-testing (manual per-tool checklists), edt-mcp-ready-to-deploy (final pre-merge checklist).
1---2name: edt-mcp-e2e-testing-23description: How to write/run the AUTOMATED black-box e2e suite (tests/e2e/) that covers every EDT-MCP tool (62 today) against a live server with git-fixture isolation, happy + negative + error-quality coverage, and an anti-cheat bar. Use when adding/editing a test in tests/e2e/, adding a new tool (the coverage ratchet requires a test), or running the suite. (For the older MANUAL per-tool reference checklists see edt-mcp-testing; for Java unit/build see edt-mcp-build-test.)4---56# EDT-MCP — automated black-box e2e suite (tests/e2e/)78> **The full, authoritative, always-current guide is `tests/e2e/SKILL.md` in the repo. READ IT before writing or running a test** — it has the harness API, the worked examples, and the anti-cheat anti-pattern list. This skill is the entrypoint + the rules in brief; the repo file wins on any disagreement (it lives with the harness).910## What this is11A Python (stdlib-only) **black-box** suite: a real MCP client hits the **live** server on `:8765` and asserts the **real** effect. One file per tool: `tests/e2e/tools/test_<tool>.py`. Shared base: `harness.py`. Runner: `run_all.py` (serial, `--junit-xml`, per-test timeout). Every registered tool covered (62 today; the coverage ratchet `test_coverage_ratchet.py` fails the suite if a `tools/list` tool has no test). Different layer from Java/JUnit (which is headless and can't touch a real project).1213## Run it14```15python tests/e2e/run_all.py --project TestConfiguration --junit-xml tests/e2e/e2e-results.xml16```17Needs a LIVE server on `:8765` + `TestConfiguration` loaded (the dev EDT copy — see the project memory / edt-redeploy loop). Execution is **serial** (one fixture, one git tree); authoring can be parallel (one agent per tool file). `--filter <substr>` runs a subset.1819## The rules in brief (details in tests/e2e/SKILL.md)201. **One file per tool**, `tests/e2e/tools/test_<tool>.py`, importing ONLY from `harness`. Adding a new MCP tool? add its test file — the **coverage ratchet** fails the suite otherwise.212. **git-fixture isolation:** `TestConfiguration` is a committed fixture. The orchestrator resets it before every test (`git reset` + `git checkout HEAD --` + `git clean -fd`). Read tools must leave it clean (`assert_no_diff()`).223. **Verify the effect the RIGHT way per `kind=`:**23 - `read` → `assert_no_diff()` + content.24 - `write` (file-write, write_module_source) → `assert_diff_contains()` (on disk).25 - `write-metadata` (create/add/delete/rename) → **MODEL READ-BACK + on-disk structure.** Call a read tool and assert the change in the model, AND assert it landed on disk via `poll_diff_contains` of the **structural element** (`<name>X</name>`, `<catalogs>Type.X</catalogs>` — not the bare name). Metadata writes now persist to disk via forceExport (~1-2s lag → poll). The orchestrator runs `reset_fixture`+`reset_model` after these.26 - `action` → usually `assert_no_diff()` + status; never run a destructive happy mutation that corrupts the fixture.27 - **env-dependent** (form-render JVM flag, LanguageTool, debug no-session) → env-robust branch: real result OR a clear actionable **sentinel**.284. **Negative coverage is MANDATORY** (not just happy): invalid path/object, every XOR/conditional/enum, missing-required, boundaries. Assert **error QUALITY** (`assert_error_quality`): names the bad value + actionable next step. A vague/non-actionable tool error → `# AUDIT:` note + a fix-card; **do NOT weaken the test** to a bad error.295. **Anti-cheat (mutation thinking):** "would this test FAIL if the tool were broken?" If not, it's a CHEAT. No trivial asserts, no `assert_no_diff()`-only, no swallowed exceptions, no tolerant matching. A separate anti-cheat reviewer rules REAL/CHEAT.3031## Don't32- Don't add pip dependencies (stdlib only; the client is kept spec-conformant by hand on purpose).33- Don't assert a metadata-write effect with the BARE name in the diff (it false-matches the parent's collection reference) — assert the structural element via `poll_diff_contains`, paired with a model read-back. (Metadata writes DO persist now, with a ~1-2s lag.)34- Don't parallelize the RUN (serial only). Don't call `reset_fixture`/`reset_model` in a test (the orchestrator owns them).3536## Protocol conformance (a separate gate)37This suite tests tool **business logic**. The MCP **protocol** itself (initialize handshake, capabilities/version negotiation, `Mcp-Session-Id`, `Accept`/`Content-Type`, `isError`, `ping`, SSE) is a different layer, checked by the official `modelcontextprotocol/conformance` suite — see `tests/conformance/` (`tests/conformance/README.md`). Run: `npx @modelcontextprotocol/conformance@latest server --url http://127.0.0.1:8765/mcp --spec-version 2025-11-25 --expected-failures tests/conformance/baseline.yml` → green when only the pinned (intentional) gaps fail. Don't mix the two gates.3839Related: `edt-mcp-tool-conventions` (param/error/output canon), `edt-mcp-bilingual` (ru/en), `edt-mcp-build-test` (Java unit/build), `edt-mcp-testing` (manual per-tool checklists), `edt-mcp-ready-to-deploy` (final pre-merge checklist).
ditrixnew