edt-mcp-e2e-testing

name	edt-mcp-e2e-testing
description	How to write and run black-box end-to-end (e2e) tests for the EDT-MCP server. Covers the architecture (shared harness + one file per tool + orchestrator), the git-fixture isolation protocol, happy-path AND negative coverage, error-quality assertions, and the anti-cheat rules. An agent that has never seen this project should be able to write a correct, non-cheating test after reading this.

EDT-MCP E2E Testing — the complete guide

Audience: an agent (or human) who has never seen this project and is asked to write or run an e2e test for one MCP tool. Read this fully before writing a single line.

---

1. What you are testing (and what you are NOT)

The EDT-MCP server is a Java plugin that runs inside a live 1C:EDT IDE. It speaks MCP (JSON-RPC 2.0 over Streamable HTTP) on http://127.0.0.1:8765/mcp and exposes ~61 tools that drive a real 1C project (read code, write code, create/delete/rename metadata, debug, profile, render forms, etc.).

You are writing black-box end-to-end tests: a Python test process that is a real MCP client. It sends real tools/call requests to the real, running server and asserts on the real responses and the real on-disk effect.

• You are NOT mocking anything. No fakes, no stubs. The server, the EDT model, and the project files are all real.
• This is a different layer from the Java/JUnit unit tests. JUnit runs headless (no live EDT) and covers pure logic / schema / argument-validation. It physically cannot test what happens against a real loaded project. That is exactly the gap these e2e tests fill.

If a tool "succeeded" in its response but the project did not actually change on disk, that is a bug — and a test that only checks the response would miss it. So we verify the on-disk truth, see §4.

---

2. Architecture (how the suite is laid out)

tests/e2e/
  SKILL.md            <- this file (the contract + the rules)
  harness.py          <- THE shared base. Import everything from here. Do NOT duplicate its logic.
  run_all.py          <- the orchestrator. Discovers and runs every test, SERIALLY, with reset between.
  tools/
    test_list_projects.py          <- one file per tool. You write/own exactly one of these.
    test_write_module_source.py
    test_create_metadata.py
    ... (one per tool)
    test_coverage_ratchet.py       <- meta: fails if a tools/list tool has no @e2e_test

• harness.py owns: the HTTP/JSON-RPC client (+ SSE framing), the git-fixture helpers, and all assertion helpers (including error-quality). You call its functions; you never re-implement them.
• tools/test_<tool>.py — one file per MCP tool. It contains the test functions for that one tool: the happy path(s) and the negative matrix. Because each agent owns one file, there are no merge conflicts when many agents work in parallel.
• run_all.py — discovers every @e2e_test function, runs them one at a time (see §3 — execution is serial), resets the fixture before each, and emits a --junit-xml report.

Golden rule of parallelism: writing the test files can be parallelized across agents. Running them cannot — every test mutates the same TestConfiguration project and the same git working tree, so the orchestrator runs them serially with a hard reset between each.

---

3. The git-fixture isolation protocol (THE core mechanism)

TestConfiguration is a git-tracked 1C project committed in this repo (TestConfiguration/). That is our test fixture and our source of on-disk truth. The protocol for every test:

1. Before the test — hard reset, never trust the previous test. The orchestrator calls reset_fixture() = git reset (unstage) + git checkout HEAD -- + git clean -fd on TestConfiguration/. ALL THREE are needed: metadata delete/rename persist to disk and can land STAGED in the index, which a plain git checkout -- (restores from the index) would NOT revert. checkout HEAD -- restores from the commit; reset unstages; clean -fd removes untracked files.
2. Run the tool (one or more call(...)).
3. Assert the effect — the assertion depends on the tool kind (see §4): on-disk git-diff (file-write), MODEL read-back (metadata-write), or empty-diff (read).
4. kind="write-metadata" tools get extra isolation (the orchestrator does this, not you): immediately after such a test it runs reset_fixture() then reset_model(). Why: these tools mutate EDT's in-memory BM model, EDT may async-autosave it to disk, and a git reset alone cannot undo an unsaved model change — reset_model() (calls clean_project, which refreshes the model from the just-reset clean disk) discards it before the next test.
5. After the whole run the final git status TestConfiguration/ MUST be empty. A run that leaves the project dirty is a failed run.

Metadata writes now persist to disk (since commit 71a48ba — fix-card metadata-writes-not-persisted-to-disk, see [[project_metadata_write_persistence]]). create_metadata / modify_metadata now forceExport the mutated top object(s) to .mdo after the write; delete_metadata/rename_metadata_object always did (via the refactoring service); write_module_source flushes .bsl synchronously. Since #406 the write tools that extend AbstractMetadataWriteTool also WAIT for that export QUEUE to drain before answering (rename_metadata_object and build_external_objects do not extend it and still lag). That wait is ordered with the export only when the SAME call submitted it. Since #408 the generic delete_metadata path submits too, but only for the CONTAINER of the deleted node (its Configuration.mdo / owner .mdo); the deleted object's OWN file cannot be resubmitted at all (EDT builds a save task by looking the FQN up in the transaction, and a deleted FQN yields none), so poll for that one. Where the call did submit, a drained queue means nothing of that call is still pending - it does NOT prove the bytes are right (EDT logs a per-file write failure and completes the computation anyway), and the wait is skipped where the export state cannot be observed. So still ASSERT the bytes; you just no longer need to wait for them. Consequence: verify a metadata-write BOTH ways — a MODEL READ-BACK (semantic: a read tool sees the change in the model) AND the on-disk structure. For a call that SUBMITTED its own export (create/modify, and the specialized delete branches) assert the disk directly (assert_disk_lacks / assert_disk_path_gone / read_disk) and do it BEFORE any other MCP round trip. Poll instead for a file whose export the call did NOT submit — for the generic delete_metadata path that is the deleted object's own file, while its container's file is asserted immediately (§ the comment in test_delete_metadata.py, and #408). rename_metadata_object / build_external_objects poll for the separate reason that they do not inherit the barrier at all. The orchestrator reverts the on-disk .mdo via reset_fixture (git) and re-syncs the model via reset_model().

---

4. How to verify "something really changed" (per tool type)

Tool kind (@e2e_test(kind=...))	Examples	Happy-path assertion
read	list_projects, read_module_source, get_metadata_details, search_in_code, find_references	response content correct AND assert_no_diff() (nothing changed on disk — the read-tool guardrail)
write (file-write)	write_module_source	assert_diff_contains(...) — the change is on disk (synchronous .bsl flush)
write-metadata	create / add / delete / rename metadata	MODEL READ-BACK + on-disk structure. (1) call(...) a read tool (get_metadata_objects / get_metadata_details / get_module_structure) and assert the change in the MODEL (object present/absent, attribute present, renamed). (2) Assert it landed on disk. When the CALL ITSELF submitted the export (create/modify, and the specialized delete branches - they call forceExportToDisk), the queue is already drained when it returns unless the export state was unobservable (§3), so assert DIRECTLY (assert_diff_contains / read_disk / assert_disk_lacks / assert_disk_path_gone) and do it BEFORE any other MCP round trip - an intervening call is time an early-returning build could use to finish, which is how the assertion stops testing anything. Poll for TWO different reasons: rename_metadata_object / build_external_objects do not extend that base at all, and the generic delete_metadata path submits only its CONTAINER's export — the deleted object's own file stays unsubmittable (see the comment in test_delete_metadata.py and #408). Assert the STRUCTURE, not just the name — e.g. "<name>E2EWeight</name>" / "<catalogs>Catalog.<name></catalogs>", never the bare name (which can false-match the parent's collection reference). For a delete (removal, not addition) check that the object file is gone AND that the collection reference is removed from Configuration.mdo; for a top-object delete assert Configuration.mdo immediately (the call submits that export since #408) and POLL for the object's own file (nobody but the refactoring can schedule its removal). For delete/rename cover preview (no mutation) + confirm. For rejected/preview calls, .

Ground truth depends on the tool: a file-write is proven by the git diff; a metadata-write by the model read-back (semantic) AND the on-disk structure (asserting the specific element - directly when the call submitted its own export, polled when it did not; see §3). Always assert WHAT changed (the structural element), never merely that a diff exists or that the bare name appears somewhere; and never use assert_no_diff() as the only assertion (it must accompany a real content/sentinel check).

---

5. Negative coverage is MANDATORY (not just happy paths)

For every tool, in addition to the happy path, cover the negative matrix:

• Invalid path / non-existent object — a modulePath / fqn / objectName that does not exist; a non-existent projectName.
• Non-existent lines / ranges — startLine/endLine out of range; an oldSource (searchReplace) that is not in the file.
• Invalid parameter combinations — every XOR branch (both / neither of modulePath vs objectName), every conditional-required (formName when moduleType=FormModule, commandName when moduleType=CommandModule, oldSource when mode=searchReplace), an invalid enum value.
• Missing required parameter.

If a tool has XOR / conditional / enum parameters and you only tested the happy path, the test is incomplete.

5.1 Assert the QUALITY of the error (not just "it failed")

A negative test must assert that the error is a good error:

• result.is_error is true (machine-detectable — the server sets it via ToolResult.error).
• The message names the invalid value — which path / object / parameter was wrong.
• The message is actionable — a mini-instruction on what to do next: which sibling tool to use (e.g. "object not found — use list_projects / get_metadata_objects to find a valid name") or how to fix the call (e.g. "modulePath and objectName are mutually exclusive — pass only one").
• It is NOT a bare "Error", a raw stack trace, or an opaque exception that leaked out of the tool.

Use assert_error_quality(err, names=[...], suggests=[...]).

5.2 The audit effect — DO NOT fudge a bad error

These negative tests double as an audit of error quality. If a tool's error is vague or not actionable, that is a finding, not something to paper over:

• Do NOT weaken your assertion to match a bad error message.
• DO record it: leave a clear # AUDIT: note in the test and report it back so it becomes a fix-card (improve the tool's error text). Raising the bar on errors is a goal of this work, not a side effect.

---

6. Anti-cheat rules — what makes a test REAL vs a CHEAT

A test that does not fail when the tool is broken is worse than no test (it gives false confidence). Apply mutation thinking to your own test: "If the tool under test were broken — returned a no-op, the wrong result, or wrote nothing — would this test FAIL?" If not, it is a cheat.

A separate anti-cheat verifier subagent will read every test and rule REAL or CHEAT. Write tests that pass it. It rejects:

1. Trivially-true asserts (assert True; assert resp is not None when it always is; asserting a substring that is always present; asserting only "the request didn't error" without checking the effect).
2. A destructive test with no real on-disk check (no git diff; asserting "diff is non-empty" instead of the specific expected change; diffing the wrong path).
3. No per-test reset (reset_fixture()) before the test → a stale diff from a previous test is counted as your own.
4. A non-destructive test without assert_no_diff() (the empty-diff guardrail is missing).
5. Self-fulfilling: reading the expected value from the same in-memory source you wrote it to, instead of going to disk/git.
6. Swallowed exceptions (try/except: pass), unconditional skip/xfail, commented-out asserts, hardcoded pass.
7. Tolerant matching that hides regressions (.* regex, case-folding, normalization that eats the real difference).
8. Coverage-gaming: a "test" that calls the tool and ignores the result just to tick the coverage ratchet.
9. No final git status clean → the run leaves the project dirty.
10. A negative test that checks only is_error / the fact of failure, but not the error content (the specific invalid value + an actionable next step).
11. Only the happy path covered where the tool has XOR / conditional / enum parameters (invalid combinations not tested).
12. A write-metadata test that verifies the effect with only a MODEL READ-BACK, or only an on-disk diff, or assert_no_diff() alone — or one whose on-disk check asserts merely the bare name (false-matches the parent's collection reference) instead of the structural element (<name>X</name>, <catalogs>Type.X</catalogs>). Metadata writes persist to disk, and a call that submitted its own export also drains that queue before answering (§3 - rename_metadata_object / build_external_objects do neither), so verify BOTH: model read-back AND an assertion on the specific on-disk element - direct when the call submitted its own export, polled when it did not (for the generic delete: its container's file is immediate, the deleted object's own file is polled). For delete/rename, also: a confirm test that doesn't read back the gone/renamed object, or a preview test that doesn't assert the object is UNCHANGED.

---

7. Harness API (the contract — call these, do not re-implement)

from harness import (
    call,                                                     # tools/call (the orchestrator does the initialize handshake once)
    assert_ok, assert_error, assert_error_quality,            # outcome + error quality
    assert_contains, assert_not_contains,                     # text content
    assert_no_diff, assert_diff_contains, assert_diff_paths,  # on-disk truth (git)
    read_disk, diff,                                          # rarely needed (prefer the asserts above)
    e2e_test, PROJECT,                                        # PROJECT == "TestConfiguration"
)
# NOTE: reset_fixture() and reset_model() also exist but the ORCHESTRATOR calls them
# for you (reset_fixture before every test; reset_fixture+reset_model after every
# write-metadata test). Do NOT call them in a test unless you genuinely need an
# intermediate reset.
#
# Gated live-infobase helpers (used ONLY by tools/test_live_roundtrip.py, see §8.1):
#   requires_live_infobase(reason)   # first line; raises E2ESkip unless EDT_MCP_LIVE_INFOBASE=1
#   parse_yaxunit_counts(text)       # YAXUnit Markdown summary -> {total,passed,failed,errors,skipped,result}
#   extract_application_id(text)     # pull applicationId out of a debug-launch handle
#   wait_until_no_running_launch(config_name=...) / any_launch_running(config_name=...)  # infobase quiet
#   terminate_all_live_launches()    # blanket all=true kill — available, but the suite prefers a
#                                    # TARGETED terminate_launch by config name (a blanket kill can
#                                    # disturb EDT's shared infobase-connection registry)
#   assert_no_substantive_diff(ctx)  # like assert_no_diff but tolerant of an EDT CRLF .mdo touch
#   TESTS_PROJECT ("<PROJECT>.tests"), LIVE_LAUNCH_CONFIG  # the extension project + the launch config

# --- calling a tool ---
r = call("write_module_source", {: PROJECT,
                                 : ,
                                 : , : })
r.is_error      
r.text          
r.structured    
r.error_text()  


assert_ok(r, ctx=)          
assert_contains(r.text, , ctx=)        
assert_diff_contains()                      
assert_no_diff()                                  


e = assert_error(r, ctx=)    
assert_error_quality(e,
    names=[],                    
    suggests=[, ])       

Test registration: each test is @e2e_test(tool="<tool_name>", kind=...). The kind picks the isolation + the verification idiom:

• kind="read" — read/list/search/nav/debug. Guardrail: assert_no_diff().
• kind="write" — file-write (write_module_source). Verify with assert_diff_contains() (on disk).
• kind="write-metadata" — create/add/delete/rename metadata. Verify with MODEL READ-BACK (call a read tool) AND on-disk structure (assert_diff_contains of the element, or assert_disk_path_gone/assert_disk_lacks for a delete branch that submits its own export; the polling variants for everything else, §3). The orchestrator runs reset_fixture()+reset_model() after each such test (you don't).
• kind="action" — clean/revalidate/import/export/update_database. Usually assert_no_diff() + status.

The orchestrator discovers @e2e_test functions, runs them serially, reset_fixture() before each, and enforces final cleanliness.

@e2e_test(tool="write_module_source", kind="write")
def test_append_adds_line_on_disk():
    r = call("write_module_source", {"projectName": "TestConfiguration",
            "modulePath": "CommonModules/OK/Module.bsl", "mode": "append",
            "source": "// e2e-probe\n"})
    assert_ok(r, "append")
    assert_diff_contains("// e2e-probe")          # proven on disk via git

@e2e_test(tool="write_module_source", kind="write")
def test_searchreplace_missing_oldsource_errors_clearly():
    r = call("write_module_source", {"projectName": "TestConfiguration",
            "modulePath": "CommonModules/OK/Module.bsl", "mode": "searchReplace",
            "oldSource": "THIS_DOES_NOT_EXIST", "source": "x"})
    e = assert_error(r, "searchReplace stale oldSource")
    assert_error_quality(e, names=["THIS_DOES_NOT_EXIST"], suggests=["read_module_source"])
    assert_no_diff()                              # a rejected write must NOT touch disk

@e2e_test(tool="create_metadata", kind="write-metadata")
def test_create_attribute_appears_in_model_and_on_disk():
    parent, new_attr = , 
    before = call(, {: PROJECT, : [parent], : })
    assert_not_contains(before.text, new_attr, )
    
    r = call(, {: PROJECT, : parent +  + new_attr})
    assert_ok(r, )
    
    
    assert_diff_contains( % new_attr, )
    after = call(, {: PROJECT, : [parent], : })
    assert_contains(after.text, new_attr, )


 ():
    r = call(, {: PROJECT})
     r.is_error:   
        assert_error_quality(r.error_text(), names=[], suggests=[])
    :            
        assert_contains(r.text, , )
    assert_no_diff()

---

8. How to run

The suite requires the live server up on :8765 with TestConfiguration loaded (the orchestrator polls /health first). Then:

python tests/e2e/run_all.py --project TestConfiguration --junit-xml tests/e2e/e2e-results.xml

• Execution within one runner is serial (see §3). Do not try to parallelize tests against a single workspace: every test resets the same git fixture and the same EDT model.
• Sharding across MACHINES is the supported way to go faster: --shard I/N runs only every Nth test (round-robin, deterministic), and each shard is a normal serial run with its own baseline, cleanup, exit code and JUnit report. CI does this with a matrix, so every shard gets its own runner, clone, EDT install and port — the isolation comes from not sharing a working tree, never from locking one. Never point two shards at the same checkout. Round-robin, not contiguous blocks, because tests are registered file by file and modify_metadata alone is ~40% of the wall clock: block-splitting caps the speed-up at ~2.5×, striping scales linearly (4 shards ≈ 3.8× on measured durations). Change the shard count in ONE place — inputs.shards in e2e-tests.yml; the divisor comes from strategy.job-total.
• The run mutates TestConfiguration and reverts it; on a clean checkout that is expected. The final state must be clean.
• The existing .github/workflows/e2e-tests.yml invokes the runner against a running server; keep its CLI flags (--host/--port/--project/--junit-xml) stable.
• No new dependencies — Python stdlib only (urllib, json, subprocess for git, re). Do not add pip packages. (Research-backed: the official MCP SDK's in-memory transport doesn't fit a server that lives inside EDT; the hand-rolled client is kept spec-conformant instead — initialize + notifications/initialized handshake, Mcp-Session-Id + MCP-Protocol-Version headers, robust SSE.)
• Scope: the suite covers every tool + tools/test_coverage_ratchet.py (fails if tools/list advertises a tool with no @e2e_test). It replaced the older run_e2e_tests.py monolith (now removed). Adding a new tool? add tools/test_<tool>.py — the ratchet enforces it.
• Filter: --filter <substr> runs only tests whose name/tool matches (useful while iterating).
• Per-test timeout / hang protection: each test runs under a wall-clock timeout (--test-timeout, default , env ; the per-CALL HTTP timeout is , default 180s). If a test exceeds it, EDT is almost certainly (the classic case: → wedges and the call never returns). That test is reported ( in JUnit) with a clear message, and — a wedged EDT would make them all hang too, so the run aborts fast instead of stalling for minutes. There is : restart EDT and re-run (optionally from where it stopped). Keep the timeout comfortably above the slowest legit test — a unit chains the test call + + (each ≤ ), so the default 600 sits well above that ~3× budget. do not start a full run right after a relaunch — let the project finish indexing first, or an early can legitimately run long.

8.1 The gated live-infobase round-trip suite (tools/test_live_roundtrip.py)

Most runtime tools (YAXUnit, debug, profiling) have no infobase/session in CI, so their per-tool files assert the realistic no-session SENTINEL (see §4 env-dependent). What a headless run cannot prove is a real end-to-end round-trip: a real YAXUnit run → parsed pass/fail counts, a real breakpoint suspend → inspect → resume, and a launch/session id minted by one tool and consumed by its siblings. Those live in one deliberately cross-tool file — the only exception to the "one file per tool" rule (the tools it touches are already covered by their own sentinel files; this file is supplementary round-trip coverage).

It is gated behind requires_live_infobase() (first line of every test): each test raises E2ESkip — reported as [SKIP], NOT a failure, and excluded from the pass/fail denominator — unless EDT_MCP_LIVE_INFOBASE=1. So a normal run_all.py runs them as skips and stays green; an attended operator runs them explicitly:

EDT_MCP_LIVE_INFOBASE=1 python tests/e2e/run_all.py --project TestConfiguration --filter test_live_

Attended preconditions (the operator owns these): EDT open on the TestConfiguration workspace + MCP up; a runtime-client launch config (default "TestConfiguration Thin Client", override MCP_LIVE_LAUNCH_CONFIG); the YAXUnit engine (YAxUnit.cfe) loaded in the infobase and the tests extension applied. The fail-count round-trip uses a dedicated tests_FailureDemo module (one passing + one deliberately failing test, kept separate so tests_SampleTests stays green) — a default no-filter YAXUnit run therefore includes one intentional failure (that is the point: it proves failures surface). Each live test quiets the infobase at both ends — a TARGETED terminate_launch on its own config name (a blanket all=true kill can disturb EDT's shared infobase-connection registry and break the next launch), plus wait_for_project_ready so a debug launch is not fired at a still-indexing project — and passes the launch's minted applicationId explicitly to wait_for_break. None touch the git source tree, but a live EDT may CRLF-touch a metadata .mdo while updating the infobase, so they end with assert_no_substantive_diff() (tolerant of a line-ending-only touch, strict on real content/structure changes).

NB this suite surfaced (and fixed, same change-set) two real tool defects a headless run never could: get_variables leaked a raw NPE on a live frame whose hasVariables() was lazy (VariableSerializer now calls getVariables() directly), and resume with no args dead-ended on a 1C target whose canResume() is false while a thread is suspended (ResumeTool now falls back to resuming the target's suspended threads).

---

9. Conventions checklist (before you say a test file is done)

• File is tests/e2e/tools/test_<tool_name>.py, imports only from harness.
• Correct kind= chosen, and the effect is verified the RIGHT way: on-disk diff (file-write) / model read-back + a direct on-disk assertion (write-metadata) / assert_no_diff() (read) / env-robust sentinel branch (env-dependent). Never assert_no_diff() as the only assertion.
• Negative matrix: invalid path/object, invalid param combinations (every XOR/conditional/enum), missing required, boundaries.
• Every negative asserts error quality (names the bad value + actionable), not just is_error.
• Any vague/non-actionable error is flagged with # AUDIT: and reported (not fudged).
• No anti-pattern from §6. Each test would FAIL if the tool were broken (mutation thinking).
• Python stdlib only; no sleep() (use the harness poll if needed); no try/except: pass.

This file is the canonical, authoritative guide for the edt-mcp-e2e-testing Claude skill (.claude/skills/edt-mcp-e2e-testing/SKILL.md is a thin entrypoint that points here). It is a living document — keep it current as the harness/tests evolve. If something here disagrees with the actual harness.py, the harness wins, and this file is corrected to match.