# Edt MCP E2e Testing > How to write and run black-box end-to-end (e2e) tests for the EDT-MCP server. Covers the architecture (shared harness + one file per tool + orchestrator), the git-fixture isolation protocol, happy-path AND negative coverage, error-quality assertions, and the anti-cheat rules. An agent that has never seen this project should be able to write a correct, non-cheating test after reading this. - Skill: `ditrixnew/edt-mcp-e2e-testing` (Agent Skill, multi-file: 131 files) - Install (CLI): `npx skillmds@latest add ditrixnew/edt-mcp-e2e-testing` - Raw SKILL.md: https://api.skillmd.com/api/skills/ditrixnew/edt-mcp-e2e-testing/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: AI & ML - Author: ditrixnew (https://skillmd.com/u/ditrixnew) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/ditrixnew/edt-mcp-e2e-testing --- # EDT-MCP E2E Testing — the complete guide > Audience: an agent (or human) who has **never** seen this project and is asked to write or run an e2e test for one MCP tool. Read this **fully** before writing a single line. --- ## 1. What you are testing (and what you are NOT) The **EDT-MCP server** is a Java plugin that runs **inside a live 1C:EDT IDE**. It speaks **MCP (JSON-RPC 2.0 over Streamable HTTP)** on `http://127.0.0.1:8765/mcp` and exposes ~61 *tools* that drive a real 1C project (read code, write code, create/delete/rename metadata, debug, profile, render forms, etc.). You are writing **black-box end-to-end tests**: a Python test process that is a **real MCP client**. It sends real `tools/call` requests to the **real, running** server and asserts on the **real** responses and the **real** on-disk effect. - You are **NOT** mocking anything. No fakes, no stubs. The server, the EDT model, and the project files are all real. - This is a **different layer** from the Java/JUnit unit tests. JUnit runs headless (no live EDT) and covers pure logic / schema / argument-validation. It physically *cannot* test what happens against a real loaded project. That is exactly the gap these e2e tests fill. If a tool "succeeded" in its response but the project did not actually change on disk, that is a **bug** — and a test that only checks the response would miss it. So we verify the **on-disk truth**, see §4. --- ## 2. Architecture (how the suite is laid out) ``` tests/e2e/ SKILL.md <- this file (the contract + the rules) harness.py <- THE shared base. Import everything from here. Do NOT duplicate its logic. run_all.py <- the orchestrator. Discovers and runs every test, SERIALLY, with reset between. tools/ test_list_projects.py <- one file per tool. You write/own exactly one of these. test_write_module_source.py test_create_metadata.py ... (one per tool) test_coverage_ratchet.py <- meta: fails if a tools/list tool has no @e2e_test ``` - **`harness.py`** owns: the HTTP/JSON-RPC client (+ SSE framing), the git-fixture helpers, and all assertion helpers (including error-quality). You call its functions; you never re-implement them. - **`tools/test_.py`** — one file per MCP tool. It contains the test functions for that one tool: the happy path(s) **and** the negative matrix. Because each agent owns one file, there are **no merge conflicts** when many agents work in parallel. - **`run_all.py`** — discovers every `@e2e_test` function, runs them **one at a time** (see §3 — execution is serial), resets the fixture before each, and emits a `--junit-xml` report. **Golden rule of parallelism:** writing the test files can be parallelized across agents. **Running** them cannot — every test mutates the same `TestConfiguration` project and the same git working tree, so the orchestrator runs them serially with a hard reset between each. --- ## 3. The git-fixture isolation protocol (THE core mechanism) `TestConfiguration` is a **git-tracked 1C project committed in this repo** (`TestConfiguration/`). That is our test fixture and our source of on-disk truth. The protocol for **every** test: 1. **Before the test — hard reset, never trust the previous test.** The orchestrator calls `reset_fixture()` = `git reset` (unstage) + `git checkout HEAD --` + `git clean -fd` on `TestConfiguration/`. ALL THREE are needed: metadata delete/rename persist to disk **and can land STAGED** in the index, which a plain `git checkout -- ` (restores from the index) would NOT revert. `checkout HEAD --` restores from the commit; `reset` unstages; `clean -fd` removes untracked files. 2. **Run the tool** (one or more `call(...)`). 3. **Assert the effect** — the assertion depends on the tool kind (see §4): on-disk git-diff (file-write), MODEL read-back (metadata-write), or empty-diff (read). 4. **`kind="write-metadata"` tools get extra isolation** (the orchestrator does this, not you): immediately after such a test it runs `reset_fixture()` **then** `reset_model()`. Why: these tools mutate EDT's in-memory BM model, EDT may **async-autosave** it to disk, and a git reset alone cannot undo an *unsaved model* change — `reset_model()` (calls `clean_project`, which refreshes the model from the just-reset clean disk) discards it before the next test. 5. **After the whole run** the final `git status TestConfiguration/` MUST be empty. A run that leaves the project dirty is a failed run. **Metadata writes now persist to disk** (since commit `71a48ba` — fix-card `metadata-writes-not-persisted-to-disk`, see [[project_metadata_write_persistence]]). `create_metadata` / `modify_metadata` now `forceExport` the mutated top object(s) to `.mdo` after the write; `delete_metadata`/`rename_metadata_object` always did (via the refactoring service); `write_module_source` flushes `.bsl` synchronously. Since #406 the write tools that extend `AbstractMetadataWriteTool` also WAIT for that export QUEUE to drain before answering (`rename_metadata_object` and `build_external_objects` do not extend it and still lag). That wait is ordered with the export only when the SAME call submitted it. Since #408 the generic `delete_metadata` path submits too, but only for the CONTAINER of the deleted node (its `Configuration.mdo` / owner `.mdo`); the deleted object's OWN file cannot be resubmitted at all (EDT builds a save task by looking the FQN up in the transaction, and a deleted FQN yields none), so poll for that one. Where the call did submit, a drained queue means nothing of that call is still pending - it does NOT prove the bytes are right (EDT logs a per-file write failure and completes the computation anyway), and the wait is skipped where the export state cannot be observed. So still ASSERT the bytes; you just no longer need to wait for them. **Consequence:** verify a metadata-write BOTH ways — a **MODEL READ-BACK** (semantic: a read tool sees the change in the model) AND the **on-disk structure**. For a call that SUBMITTED its own export (create/modify, and the specialized delete branches) assert the disk directly (`assert_disk_lacks` / `assert_disk_path_gone` / `read_disk`) and do it BEFORE any other MCP round trip. Poll instead for a file whose export the call did NOT submit — for the generic `delete_metadata` path that is the deleted object's own file, while its container's file is asserted immediately (§ the comment in `test_delete_metadata.py`, and #408). `rename_metadata_object` / `build_external_objects` poll for the separate reason that they do not inherit the barrier at all. The orchestrator reverts the on-disk `.mdo` via `reset_fixture` (git) and re-syncs the model via `reset_model()`. --- ## 4. How to verify "something really changed" (per tool type) | Tool kind (`@e2e_test(kind=...)`) | Examples | Happy-path assertion | |---|---|---| | **read** | list_projects, read_module_source, get_metadata_details, search_in_code, find_references | response content correct **AND** `assert_no_diff()` (nothing changed on disk — the read-tool guardrail) | | **write** (file-write) | write_module_source | `assert_diff_contains(...)` — the change is on disk (synchronous `.bsl` flush) | | **write-metadata** | create / add / delete / rename metadata | **MODEL READ-BACK + on-disk structure.** (1) `call(...)` a read tool (get_metadata_objects / get_metadata_details / get_module_structure) and assert the change in the MODEL (object present/absent, attribute present, renamed). (2) Assert it landed on disk. When the CALL ITSELF submitted the export (create/modify, and the specialized delete branches - they call `forceExportToDisk`), the queue is already drained when it returns unless the export state was unobservable (§3), so assert DIRECTLY (`assert_diff_contains` / `read_disk` / `assert_disk_lacks` / `assert_disk_path_gone`) and do it BEFORE any other MCP round trip - an intervening call is time an early-returning build could use to finish, which is how the assertion stops testing anything. Poll for TWO different reasons: `rename_metadata_object` / `build_external_objects` do not extend that base at all, and the generic `delete_metadata` path submits only its CONTAINER's export — the deleted object's own file stays unsubmittable (see the comment in `test_delete_metadata.py` and #408). **Assert the STRUCTURE, not just the name** — e.g. `"E2EWeight"` / `"Catalog."`, never the bare name (which can false-match the parent's collection reference). For a **delete** (removal, not addition) check that the object file is gone AND that the collection reference is removed from `Configuration.mdo`; for a top-object delete assert `Configuration.mdo` immediately (the call submits that export since #408) and POLL for the object's own file (nobody but the refactoring can schedule its removal). For delete/rename cover preview (no mutation) + confirm. For rejected/preview calls, `assert_no_diff()`. | | **action** | clean_project, revalidate_objects, import/export_configuration, update_database | usually `assert_no_diff()` (they touch validation markers / the infobase DB / an external XML dump, NOT project files) + assert the status. Do NOT run a destructive happy mutation (import/update_database) that could corrupt the fixture — cover the error/sentinel matrix. | | **env-dependent** | get_form_screenshot / get_form_layout_snapshot (JVM render flag), translate_configuration / generate_translation_strings / get_translation_project_info (LanguageTool plugin), debug/profiling (no session/infobase in this env) | **env-robust branch:** assert the REAL observed contract — a structured result IF the dependency is present, OR a clear, actionable **SENTINEL** ("LanguageTool ... not available. Install ...", "no active debug session") if not. Both branches must be mutation-sensitive (a wrong/blank payload fails). | | **round-trip ID** | get_applications → launch, find_references → read_* | assert the returned ID is consumable by the sibling tool | **Ground truth depends on the tool:** a **file-write** is proven by the git diff; a **metadata-write** by the model read-back (semantic) AND the on-disk structure (asserting the specific element - directly when the call submitted its own export, polled when it did not; see §3). Always assert WHAT changed (the structural element), never merely that *a* diff exists or that the bare name appears somewhere; and never use `assert_no_diff()` as the *only* assertion (it must accompany a real content/sentinel check). --- ## 5. Negative coverage is MANDATORY (not just happy paths) For **every** tool, in addition to the happy path, cover the **negative matrix**: - **Invalid path / non-existent object** — a `modulePath` / `fqn` / `objectName` that does not exist; a non-existent `projectName`. - **Non-existent lines / ranges** — `startLine`/`endLine` out of range; an `oldSource` (searchReplace) that is not in the file. - **Invalid parameter combinations** — every XOR branch (both / neither of `modulePath` vs `objectName`), every conditional-required (`formName` when `moduleType=FormModule`, `commandName` when `moduleType=CommandModule`, `oldSource` when `mode=searchReplace`), an invalid `enum` value. - **Missing required parameter.** If a tool has XOR / conditional / enum parameters and you only tested the happy path, the test is **incomplete**. ### 5.1 Assert the QUALITY of the error (not just "it failed") A negative test must assert that the error is a **good** error: - `result.is_error` is `true` (machine-detectable — the server sets it via `ToolResult.error`). - The message **names the invalid value** — *which* path / object / parameter was wrong. - The message is **actionable** — a mini-instruction on what to do next: which sibling tool to use (e.g. *"object not found — use list_projects / get_metadata_objects to find a valid name"*) or how to fix the call (e.g. *"modulePath and objectName are mutually exclusive — pass only one"*). - It is **NOT** a bare `"Error"`, a raw stack trace, or an opaque exception that leaked out of the tool. Use `assert_error_quality(err, names=[...], suggests=[...])`. ### 5.2 The audit effect — DO NOT fudge a bad error These negative tests double as an **audit of error quality**. If a tool's error is vague or not actionable, that is a **finding**, not something to paper over: - **Do NOT** weaken your assertion to match a bad error message. - **DO** record it: leave a clear `# AUDIT:` note in the test and report it back so it becomes a **fix-card** (improve the tool's error text). Raising the bar on errors is a goal of this work, not a side effect. --- ## 6. Anti-cheat rules — what makes a test REAL vs a CHEAT A test that does not fail when the tool is broken is **worse than no test** (it gives false confidence). Apply **mutation thinking** to your own test: *"If the tool under test were broken — returned a no-op, the wrong result, or wrote nothing — would this test FAIL?"* If not, it is a cheat. A separate **anti-cheat verifier subagent** will read every test and rule REAL or CHEAT. Write tests that pass it. It rejects: 1. Trivially-true asserts (`assert True`; `assert resp is not None` when it always is; asserting a substring that is always present; asserting only "the request didn't error" without checking the effect). 2. A destructive test with **no real on-disk check** (no `git diff`; asserting "diff is non-empty" instead of the **specific** expected change; diffing the wrong path). 3. No per-test reset (`reset_fixture()`) before the test → a stale diff from a previous test is counted as your own. 4. A non-destructive test without `assert_no_diff()` (the empty-diff guardrail is missing). 5. Self-fulfilling: reading the expected value from the same in-memory source you wrote it to, instead of going to disk/git. 6. Swallowed exceptions (`try/except: pass`), unconditional `skip`/`xfail`, commented-out asserts, hardcoded pass. 7. Tolerant matching that hides regressions (`.*` regex, case-folding, normalization that eats the real difference). 8. Coverage-gaming: a "test" that calls the tool and ignores the result just to tick the coverage ratchet. 9. No final `git status` clean → the run leaves the project dirty. 10. A negative test that checks only `is_error` / the fact of failure, but **not the error content** (the specific invalid value + an actionable next step). 11. Only the happy path covered where the tool has XOR / conditional / enum parameters (invalid combinations not tested). 12. A `write-metadata` test that verifies the effect with **only** a MODEL READ-BACK, or **only** an on-disk diff, or `assert_no_diff()` alone — or one whose on-disk check asserts merely the **bare name** (false-matches the parent's collection reference) instead of the **structural element** (`X`, `Type.X`). Metadata writes persist to disk, and a call that submitted its own export also drains that queue before answering (§3 - `rename_metadata_object` / `build_external_objects` do neither), so verify BOTH: model read-back AND an assertion on the specific on-disk element - direct when the call submitted its own export, polled when it did not (for the generic delete: its container's file is immediate, the deleted object's own file is polled). For delete/rename, also: a confirm test that doesn't read back the gone/renamed object, or a preview test that doesn't assert the object is UNCHANGED. --- ## 7. Harness API (the contract — call these, do not re-implement) ```python from harness import ( call, # tools/call (the orchestrator does the initialize handshake once) assert_ok, assert_error, assert_error_quality, # outcome + error quality assert_contains, assert_not_contains, # text content assert_no_diff, assert_diff_contains, assert_diff_paths, # on-disk truth (git) read_disk, diff, # rarely needed (prefer the asserts above) e2e_test, PROJECT, # PROJECT == "TestConfiguration" ) # NOTE: reset_fixture() and reset_model() also exist but the ORCHESTRATOR calls them # for you (reset_fixture before every test; reset_fixture+reset_model after every # write-metadata test). Do NOT call them in a test unless you genuinely need an # intermediate reset. # # Gated live-infobase helpers (used ONLY by tools/test_live_roundtrip.py, see §8.1): # requires_live_infobase(reason) # first line; raises E2ESkip unless EDT_MCP_LIVE_INFOBASE=1 # parse_yaxunit_counts(text) # YAXUnit Markdown summary -> {total,passed,failed,errors,skipped,result} # extract_application_id(text) # pull applicationId out of a debug-launch handle # wait_until_no_running_launch(config_name=...) / any_launch_running(config_name=...) # infobase quiet # terminate_all_live_launches() # blanket all=true kill — available, but the suite prefers a # # TARGETED terminate_launch by config name (a blanket kill can # # disturb EDT's shared infobase-connection registry) # assert_no_substantive_diff(ctx) # like assert_no_diff but tolerant of an EDT CRLF .mdo touch # TESTS_PROJECT (".tests"), LIVE_LAUNCH_CONFIG # the extension project + the launch config # --- calling a tool --- r = call("write_module_source", {"projectName": PROJECT, "modulePath": "CommonModules/OK/Module.bsl", "mode": "append", "source": "// x\n"}) r.is_error # bool - did the tool report an error (server sets isError) r.text # str - content[0].text (digest/markdown). JSON tools put data in .structured! r.structured # dict|None - structuredContent (None for MARKDOWN/TEXT tools; the data for JSON tools) r.error_text() # str - best-effort error message (structured.error, then text) # --- happy-path assertions --- assert_ok(r, ctx="append to OK module") # fails if r.is_error assert_contains(r.text, "...", ctx="...") # / assert_not_contains(...) assert_diff_contains("// x") # file-write: the on-disk diff includes this assert_no_diff() # read/rejected: working tree for TestConfiguration is clean # --- negative + error-quality --- e = assert_error(r, ctx="missing projectName") # asserts is_error; returns the error text assert_error_quality(e, names=["OK/DoesNotExist"], # the message names the bad value suggests=["list_modules", "modulePath"]) # the message is actionable (mentions a next step/tool) ``` **Test registration:** each test is `@e2e_test(tool="", kind=...)`. The **kind** picks the isolation + the verification idiom: - `kind="read"` — read/list/search/nav/debug. Guardrail: `assert_no_diff()`. - `kind="write"` — file-write (write_module_source). Verify with `assert_diff_contains()` (on disk). - `kind="write-metadata"` — create/add/delete/rename metadata. Verify with MODEL READ-BACK (call a read tool) AND on-disk structure (`assert_diff_contains` of the element, or `assert_disk_path_gone`/`assert_disk_lacks` for a delete branch that submits its own export; the polling variants for everything else, §3). The orchestrator runs `reset_fixture()`+`reset_model()` after each such test (you don't). - `kind="action"` — clean/revalidate/import/export/update_database. Usually `assert_no_diff()` + status. The orchestrator discovers `@e2e_test` functions, runs them serially, `reset_fixture()` **before each**, and enforces final cleanliness. ```python @e2e_test(tool="write_module_source", kind="write") def test_append_adds_line_on_disk(): r = call("write_module_source", {"projectName": "TestConfiguration", "modulePath": "CommonModules/OK/Module.bsl", "mode": "append", "source": "// e2e-probe\n"}) assert_ok(r, "append") assert_diff_contains("// e2e-probe") # proven on disk via git @e2e_test(tool="write_module_source", kind="write") def test_searchreplace_missing_oldsource_errors_clearly(): r = call("write_module_source", {"projectName": "TestConfiguration", "modulePath": "CommonModules/OK/Module.bsl", "mode": "searchReplace", "oldSource": "THIS_DOES_NOT_EXIST", "source": "x"}) e = assert_error(r, "searchReplace stale oldSource") assert_error_quality(e, names=["THIS_DOES_NOT_EXIST"], suggests=["read_module_source"]) assert_no_diff() # a rejected write must NOT touch disk @e2e_test(tool="create_metadata", kind="write-metadata") def test_create_attribute_appears_in_model_and_on_disk(): parent, new_attr = "Catalog.Catalog", "E2EColor" before = call("get_metadata_details", {"projectName": PROJECT, "objectFqns": [parent], "full": True}) assert_not_contains(before.text, new_attr, "absent before (else a no-op passes)") # Members are addressed by their full FQN (the create_metadata fold of add_metadata_attribute). r = call("create_metadata", {"projectName": PROJECT, "fqn": parent + ".Attribute." + new_attr}) assert_ok(r, "create attribute member") # DISK FIRST: create submits its own export, so the bytes are settled the moment the call # returns - but only until the next MCP round trip gives a slower build time to catch up. assert_diff_contains("%s" % new_attr, "ON-DISK: the new attribute reaches the .mdo") after = call("get_metadata_details", {"projectName": PROJECT, "objectFqns": [parent], "full": True}) assert_contains(after.text, new_attr, "MODEL read-back: new attribute is in the model") @e2e_test(tool="get_translation_project_info", kind="read") def test_env_dependent_sentinel_or_real(): r = call("get_translation_project_info", {"projectName": PROJECT}) if r.is_error: # LanguageTool plugin absent in this EDT -> a clear, actionable sentinel assert_error_quality(r.error_text(), names=["LanguageTool"], suggests=["Install"]) else: # plugin present -> the real structured contract assert_contains(r.text, "## Storages", "translation info document") assert_no_diff() ``` --- ## 8. How to run The suite requires the **live server up** on `:8765` with `TestConfiguration` loaded (the orchestrator polls `/health` first). Then: ``` python tests/e2e/run_all.py --project TestConfiguration --junit-xml tests/e2e/e2e-results.xml ``` - Execution within one runner is **serial** (see §3). Do not try to parallelize tests against a single workspace: every test resets the same git fixture and the same EDT model. - **Sharding across MACHINES is the supported way to go faster:** `--shard I/N` runs only every Nth test (round-robin, deterministic), and each shard is a normal serial run with its own baseline, cleanup, exit code and JUnit report. CI does this with a matrix, so every shard gets its own runner, clone, EDT install and port — the isolation comes from **not sharing a working tree**, never from locking one. **Never point two shards at the same checkout.** Round-robin, not contiguous blocks, because tests are registered file by file and `modify_metadata` alone is ~40% of the wall clock: block-splitting caps the speed-up at ~2.5×, striping scales linearly (4 shards ≈ 3.8× on measured durations). Change the shard count in ONE place — `inputs.shards` in `e2e-tests.yml`; the divisor comes from `strategy.job-total`. - The run **mutates** `TestConfiguration` and reverts it; on a clean checkout that is expected. The final state must be clean. - The existing `.github/workflows/e2e-tests.yml` invokes the runner against a running server; keep its CLI flags (`--host/--port/--project/--junit-xml`) stable. - **No new dependencies** — Python **stdlib only** (`urllib`, `json`, `subprocess` for git, `re`). Do not add pip packages. (Research-backed: the official MCP SDK's in-memory transport doesn't fit a server that lives inside EDT; the hand-rolled client is kept spec-conformant instead — initialize + notifications/initialized handshake, Mcp-Session-Id + MCP-Protocol-Version headers, robust SSE.) - **Scope:** the suite covers **every tool** + `tools/test_coverage_ratchet.py` (fails if `tools/list` advertises a tool with no `@e2e_test`). It replaced the older `run_e2e_tests.py` monolith (now removed). **Adding a new tool?** add `tools/test_.py` — the ratchet enforces it. - **Filter:** `--filter ` runs only tests whose name/tool matches (useful while iterating). - **Per-test timeout / hang protection:** each test runs under a wall-clock timeout (`--test-timeout`, default **600s**, env `MCP_TEST_TIMEOUT`; the per-CALL HTTP timeout is `MCP_CALL_TIMEOUT`, default 180s). If a test exceeds it, EDT is almost certainly **hung** (the classic case: `clean_project` → `ProjectRestartJob` wedges and the call never returns). That test is reported **`[TIMEOUT]` = a FAILURE** (`` in JUnit) with a clear message, and **every remaining test is `[SKIP]`ped** — a wedged EDT would make them all hang too, so the run aborts fast instead of stalling for minutes. There is **no auto-relaunch**: restart EDT and re-run (optionally `--filter` from where it stopped). Keep the timeout comfortably above the slowest legit test — a `write-metadata` unit chains the test call + `clean_project` + `wait_for_project_ready` (each ≤ `MCP_CALL_TIMEOUT`), so the default 600 sits well above that ~3× budget. **Gotcha:** do not start a full run right after a `-clean` relaunch — let the project finish indexing first, or an early `clean_project` can legitimately run long. - **Clean tree at session end (`final_cleanup`):** a run STARTS and ENDS with `harness.final_cleanup()` so `git status` is empty afterward — "no diff" is the visible "the session passed and left nothing behind" signal. It reverts **BOTH** fixtures (base `tests/TestConfiguration` + extension `tests/tests`) and `clean_project`s both so EDT's in-memory model is re-synced from the clean disk; without that model sync, a stale model (a manual edit made in the EDT editor, or an unflushed metadata write) would **autosave the change back after the run** and silently re-dirty the tree. The final cleanliness gate is `all_fixtures_status()` (covers both fixtures). NB it WILL wipe an uncommitted manual experiment under either fixture — that is intentional (a session must end clean). On a timeout-abort it reverts git-only (EDT is wedged; model-sync would hang), so post-abort cleanliness is point-in-time only. **Per-test** `assert_no_diff` stays base-scoped (tests only mutate the base, read the extension) — if you ever add an extension-MUTATING tool, give it its own extension-scoped diff assertion, since the per-test gate won't see extension dirt (only the final gate will). ### 8.1 The gated live-infobase round-trip suite (`tools/test_live_roundtrip.py`) Most runtime tools (YAXUnit, debug, profiling) have **no infobase/session in CI**, so their per-tool files assert the realistic *no-session SENTINEL* (see §4 env-dependent). What a headless run **cannot** prove is a real **end-to-end round-trip**: a real YAXUnit run → parsed pass/fail counts, a real breakpoint **suspend → inspect → resume**, and a launch/session **id minted by one tool and consumed by its siblings**. Those live in **one deliberately cross-tool file** — the only exception to the "one file per tool" rule (the tools it touches are already covered by their own sentinel files; this file is *supplementary* round-trip coverage). It is **gated** behind `requires_live_infobase()` (first line of every test): each test raises `E2ESkip` — reported as **`[SKIP]`, NOT a failure**, and excluded from the pass/fail denominator — unless `EDT_MCP_LIVE_INFOBASE=1`. So a normal `run_all.py` runs them as skips and stays green; an **attended** operator runs them explicitly: ``` EDT_MCP_LIVE_INFOBASE=1 python tests/e2e/run_all.py --project TestConfiguration --filter test_live_ ``` Attended preconditions (the operator owns these): EDT open on the TestConfiguration workspace + MCP up; a runtime-client launch config (default `"TestConfiguration Thin Client"`, override `MCP_LIVE_LAUNCH_CONFIG`); the **YAXUnit engine (`YAxUnit.cfe`) loaded** in the infobase and the `tests` extension applied. The fail-count round-trip uses a dedicated **`tests_FailureDemo`** module (one passing + one *deliberately failing* test, kept separate so `tests_SampleTests` stays green) — a default no-filter YAXUnit run therefore includes one **intentional** failure (that is the point: it proves failures surface). Each live test quiets the infobase at **both** ends — a TARGETED `terminate_launch` on its own config name (a blanket `all=true` kill can disturb EDT's shared infobase-connection registry and break the next launch), plus `wait_for_project_ready` so a debug launch is not fired at a still-indexing project — and passes the launch's minted `applicationId` explicitly to `wait_for_break`. None touch the git source tree, but a live EDT may CRLF-touch a metadata `.mdo` while updating the infobase, so they end with `assert_no_substantive_diff()` (tolerant of a line-ending-only touch, strict on real content/structure changes). NB this suite surfaced (and fixed, same change-set) two real tool defects a headless run never could: `get_variables` leaked a raw NPE on a live frame whose `hasVariables()` was lazy (`VariableSerializer` now calls `getVariables()` directly), and `resume` with no args dead-ended on a 1C target whose `canResume()` is false while a thread is suspended (`ResumeTool` now falls back to resuming the target's suspended threads). --- ## 9. Conventions checklist (before you say a test file is done) - [ ] File is `tests/e2e/tools/test_.py`, imports only from `harness`. - [ ] Correct `kind=` chosen, and the effect is verified the RIGHT way: **on-disk diff** (file-write) / **model read-back + a direct on-disk assertion** (write-metadata) / **`assert_no_diff()`** (read) / **env-robust sentinel branch** (env-dependent). Never `assert_no_diff()` as the only assertion. - [ ] Negative matrix: invalid path/object, invalid param combinations (every XOR/conditional/enum), missing required, boundaries. - [ ] Every negative asserts **error quality** (names the bad value + actionable), not just `is_error`. - [ ] Any vague/non-actionable error is flagged with `# AUDIT:` and reported (not fudged). - [ ] No anti-pattern from §6. Each test would FAIL if the tool were broken (mutation thinking). - [ ] Python stdlib only; no `sleep()` (use the harness poll if needed); no `try/except: pass`. > This file is the **canonical, authoritative guide** for the `edt-mcp-e2e-testing` Claude skill (`.claude/skills/edt-mcp-e2e-testing/SKILL.md` is a thin entrypoint that points here). It is a living document — keep it current as the harness/tests evolve. If something here disagrees with the actual `harness.py`, the harness wins, and this file is corrected to match.