field-test

## Context Unit tests verify handler logic with mocked context. Field testing exercises the real HTTP transport with real JSON-RPC: starts the server, calls `initialize`, surfaces the catalog, runs inputs, and checks what a client actually sees. It catches what unit tests miss — awkward input shapes, unhelpful errors, missing `responseFormatter` output, drift between `structuredContent` and `content[]`, edge-case surprises, and git-specific behavior around conflicts, bare repos, protected branches, working directory state, etc. **Actively call the tools. Don't read code and guess.** For git tools specifically: prefer a throwaway test repo (`/tmp/field-test-repo` initialized with `git init` and a seed commit) over the project itself so state changes are bounded and easy to clean up. --- ## Steps ### 1. Start the server Write the helper to `/tmp/mcp-field-test.sh` once, then source it in every subsequent Bash call. Helper keeps PID / URL / session id in `/tmp/mcp-field-test.env` so state survives across tool invocations. ```bash cat > /tmp/mcp-field-test.sh <<'HELPER_EOF' #!/bin/bash # Field-test helper: manage an MCP HTTP server + JSON-RPC session across shell calls. STATE_FILE="/tmp/mcp-field-test.env" [ -f "$STATE_FILE" ] && . "$STATE_FILE" mcp_start() { local dir="${1:-$PWD}" echo "building $dir ..." (cd "$dir" && bun run rebuild) >/tmp/mcp-build.log 2>&1 \ || { echo "BUILD FAILED — see /tmp/mcp-build.log"; return 1; } echo "starting server ..." (cd "$dir" && bun run start:http) >/tmp/mcp-server.log 2>&1 & local pid=$! local line="" for _ in $(seq 1 40); do line=$(grep -Eo 'listening at http://[^" ]+/mcp' /tmp/mcp-server.log | head -1) [ -n "$line" ] && break sleep 0.25 done if [ -z "$line" ]; then echo "server failed to start — see /tmp/mcp-server.log" kill "$pid" 2>/dev/null return 1 fi local url="${line#listening at }" local port; port=$(echo "$url" | sed -E 's|.*:([0-9]+)/.*|\1|') cat > "$STATE_FILE" <<EOF export MCP_PID=$pid export MCP_URL=$url export MCP_PORT=$port EOF . "$STATE_FILE" echo "ready pid=$pid url=$url" } mcp_init() { [ -z "$MCP_URL" ] && { echo "run mcp_start first"; return 1; } local hdr="/tmp/mcp-init-headers.txt" curl -sS -D "$hdr" -X POST "$MCP_URL" \ -H "Content-Type: application/json" \ -H "Accept: application/json, text/event-stream" \ -d '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-06-18","capabilities":{},"clientInfo":{"name":"field-test","version":"1.0"}}}' >/dev/null local sid; sid=$(grep -i '^mcp-session-id:' "$hdr" | awk '{print $2}' | tr -d '\r\n') [ -z "$sid" ] && { echo "no session id returned"; return 1; } cat > "$STATE_FILE" <<EOF export MCP_PID=$MCP_PID export MCP_URL=$MCP_URL export MCP_PORT=$MCP_PORT export MCP_SID=$sid EOF . "$STATE_FILE" curl -sS -X POST "$MCP_URL" \ -H "Content-Type: application/json" \ -H "Accept: application/json, text/event-stream" \ -H "Mcp-Session-Id: $sid" \ -d '{"jsonrpc":"2.0","method":"notifications/initialized"}' >/dev/null echo "session=$sid" } # Usage: mcp_call METHOD [JSON_PARAMS] # Prints the JSON-RPC response (SSE framing stripped). Pipe to `jq`. mcp_call() { [ -z "$MCP_SID" ] && { echo "run mcp_init first"; return 1; } local method="$1"; local params="${2:-}" local body if [ -z "$params" ]; then body=$(printf '{"jsonrpc":"2.0","id":%d,"method":"%s"}' "$RANDOM" "$method") else body=$(printf '{"jsonrpc":"2.0","id":%d,"method":"%s","params":%s}' "$RANDOM" "$method" "$params") fi curl -sS -X POST "$MCP_URL" \ -H "Content-Type: application/json" \ -H "Accept: application/json, text/event-stream" \ -H "Mcp-Session-Id: $MCP_SID" \ -d "$body" | sed -n 's/^data: //p' } mcp_stop() { [ -n "$MCP_PID" ] && kill "$MCP_PID" 2>/dev/null rm -f "$STATE_FILE" echo "stopped" } HELPER_EOF . /tmp/mcp-field-test.sh mcp_start /Users/casey/Developer/github/git-mcp-server ``` **Notes** - `MCP_HTTP_PORT` is a _starting_ port — the server auto-increments if taken. Helper parses the real URL from the log (`HTTP transport listening at ...`). - If `bun run rebuild` fails, stop. Don't field-test broken code — fix the build first. - If a server is already listening on the project's port (`lsof -i :<port>`), confirm with the user before killing it; it may be their own session. - `MCP_AUTH_MODE=none` is the default dev configuration — no auth headers needed. ### 2. Initialize the session ```bash . /tmp/mcp-field-test.sh mcp_init ``` Runs `initialize`, captures the session id, sends `notifications/initialized`. ### 3. Prepare a test repository Most git tools require a working directory. Create a sandboxed repo once: ```bash TEST_REPO=/tmp/field-test-repo rm -rf "$TEST_REPO" && mkdir -p "$TEST_REPO" (cd "$TEST_REPO" && \ git init -q && \ git config user.email "field-test@example.com" && \ git config user.name "Field Test" && \ echo "initial" > README.md && \ git add README.md && \ git commit -q -m "initial commit") echo "test repo ready at $TEST_REPO" ``` Pass `$TEST_REPO` as the `path` argument in tool calls (never `.`, since that resolves to session state). For tests that don't need a repo (e.g. `git_clone`, `git_set_working_dir`), use `/tmp/field-test-dest` or similar. ### 4. Surface the catalog ```bash . /tmp/mcp-field-test.sh mcp_call tools/list | jq '.result.tools[] | {name, description, inputSchema}' mcp_call resources/list | jq '.result.resources[] | {uri, name, mimeType}' mcp_call prompts/list | jq '.result.prompts[] | {name, description, arguments}' ``` Present a compact catalog to the user: each definition's name + 1-line description. Flag vague or missing descriptions as you go — those feed into the report. ### 5. Plan the test pass **Budget.** This server exposes ~29 git tools. Don't run every category against every tool — the cross-product is infeasible. Apply the **universal battery** to everything; apply **situational categories** only when the tool triggers them. **Universal battery — run on every tool** | Category | What to verify | | :--------------------------------------- | :------------------------------------------------------------------------------------------------------------------------ | | Happy path | One realistic input against `$TEST_REPO`. Output shape matches `outputSchema`. `content[]` text reads clearly to a human. | | `structuredContent` ↔ `content[]` parity | Every field in `structuredContent` is surfaced in the text. Parity gap = client-specific blindness. | | Input error | One invalid input (wrong type or missing required). Error text says _what_, _why_, _how to fix_. | **Situational — add only when triggered** | Trigger | Add category | | :--------------------------------------------------------------------------------------- | :---------------------------------------------------------------------------------------------------------- | | `verbosity` / `fields` / `include` parameter | Field selection: verify `minimal` / `standard` / `full` actually differ | | Array return with `query` / `filter` inputs (e.g. `git_log`) | Empty result: does response explain _why_ (echo criteria, suggest broadening)? | | Destructive flags (`--hard`, `--force`, `git_clean`) | Confirm guard: does the tool require explicit flag to proceed? | | Conflict-producing operations (`git_merge`, `git_rebase`, `git_cherry_pick`, `git_pull`) | Construct a conflict on purpose; confirm `{ success: true, conflicts: true, conflictedFiles: [...] }` shape | | Requires remote (`git_fetch`, `git_pull`, `git_push`, `git_clone`) | Either point at a local bare repo file:// URL or skip with a note | | Bare-repo-only behavior (`git_init --bare`) | Verify `isBare: true` surfaces; verify `git_add`/`git_commit` reject | | `annotations.readOnlyHint: true` | Confirm no mutation happened | | Session working directory (`path: "."`) | Run `git_set_working_dir`, then a read-only tool with `path: "."` to confirm resolution | | Chained workflows (status → add → commit → log) | Run one representative chain end-to-end | **Resources.** `git-working-directory.resource.ts`: happy path, unset state (before `git_set_working_dir` is called). **Prompts.** `git-wrapup.prompt.ts`: happy path, skim message quality. **Sampling strategy.** Run the universal battery on all 29 tools, but pick roughly 30–40% for situational testing. Weight toward: write-shaped tools, tools with complex schemas, tools that integrate multiple operations (`git_merge`, `git_rebase`, `git_stash`, `git_worktree`). List which ones you skipped in the report. **External state.** - `git_push` / `git_clone` against real remotes: use `file://` URLs to local bare repos, or skip with a note. - Tools that write to real external systems should never touch the user's real `origin`. ### 6. Execute Use `TaskCreate` — one task per tool (or logical group). Mark complete as you go. Don't batch. For each call, capture: input sent, response (trim huge payloads to files), whether `isError: true` appeared, anything surprising (slow response, parity drift, unhelpful text, crash). **Interpreting responses**