field-test

## Context Unit tests (`add-test` skill) verify handler logic with mocked context. Field testing exercises the real HTTP transport with real JSON-RPC: starts the server, calls `initialize`, surfaces the catalog, runs inputs, and checks what a client actually sees. It catches what unit tests miss — awkward input shapes, unhelpful errors, missing format output, drift between `structuredContent` and `content[]`, edge-case surprises. **Actively call the tools. Don't read code and guess.** ### Transport coverage This skill drives an HTTP server because curl + JSON-RPC is the most reliable harness for shell-based agents. The same handlers run on both transports — only the framing differs — so HTTP exercises the full functional surface. **Stdio coverage is a boot check only — run this before Step 1.** Run `bun run rebuild && bun run start:stdio`, confirm the startup logs look clean (banner, expected tool/resource counts, no errors/warnings, no missing-config gripes), then kill it. Pino logs go to stderr in stdio mode (stdout is reserved for JSON-RPC), so they print straight to the terminal when you run interactively. No need to call tools over stdio — the HTTP pass already covered handler behavior. --- ## Steps ### 1. Start the server Generate a 10-character alphanumeric ID (e.g. `9DJ73-K103L`) and write the helper to `/tmp/<project-name>-field-test-<ID>.sh`. Use that exact path in every subsequent Bash call. **Two agents in the same project tree must pick different IDs** — that's what keeps their helper files, server logs, and call scratch from colliding. The helper itself is **stateless** — every function takes the IDs it needs (server `pid`, `url`, `port`, MCP `sid`, server log path) as positional args. `mcp_start` prints them; the agent threads them through every later call. No env vars, no shared state files. ```bash # Pick your ID — example below uses 9DJ73-K103L. Substitute your own. # (Helper path also encodes the project name so /tmp/ stays grep-friendly.) cat > /tmp/<project-name>-field-test-9DJ73-K103L.sh <<'HELPER_EOF' #!/bin/bash # Field-test helper: stateless wrappers around an MCP HTTP server + JSON-RPC # session. Every function takes the IDs it needs as positional args — the agent # threads pid/url/port/sid/log through each call rather than relying on a state # file or env vars (the Bash tool wipes shell state between calls, and a # pointer file would race the same way two agents race on shared state). # See https://github.com/cyanheads/mcp-ts-core/issues/90, #144. # # Surfaces failures aggressively — field test is for finding things that fail, # so the helper auto-tails logs and prints HTTP status/body on errors instead # of swallowing them. # Usage: mcp_start /path/to/server [startup-timeout-seconds] (default: 30) # Builds, starts the HTTP server in the background, waits for the listen line, # and prints: ready pid=<n> url=<u> port=<n> log=<path> # Capture these — every later helper takes them as args. Raise the timeout for # servers that build a local index at boot. mcp_start() { local dir="${1:-$PWD}" local timeout="${2:-30}" local build_log; build_log=$(mktemp /tmp/mcp-field-test-build.XXXXXX) echo "building $dir ..." >&2 if ! (cd "$dir" && bun run rebuild) >"$build_log" 2>&1; then echo "BUILD FAILED — last 30 lines of $build_log:" >&2 tail -30 "$build_log" >&2 return 1 fi rm -f "$build_log" local server_log; server_log=$(mktemp /tmp/mcp-field-test-server.XXXXXX) echo "starting server ..." >&2 (cd "$dir" && bun run start:http) >"$server_log" 2>&1 & local pid=$! local line="" local waited=0 while [ "$waited" -lt "$((timeout * 4))" ]; do line=$(grep -Eo 'listening at http://[^" ]+/mcp' "$server_log" | head -1) [ -n "$line" ] && break if ! kill -0 "$pid" 2>/dev/null; then echo "server exited during startup — last 30 lines of $server_log:" >&2 tail -30 "$server_log" >&2 rm -f "$server_log" return 1 fi sleep 0.25 waited=$((waited + 1)) done if [ -z "$line" ]; then echo "server failed to start within ${timeout}s — last 30 lines of $server_log:" >&2 tail -30 "$server_log" >&2 kill "$pid" 2>/dev/null rm -f "$server_log" return 1 fi local url="${line#listening at }" local port; port=$(echo "$url" | sed -E 's|.*:([0-9]+)/.*|\1|') echo "ready pid=$pid url=$url port=$port log=$server_log" } # Usage: mcp_init <url> # Runs `initialize`, sends `notifications/initialized`, prints: # ready sid=<id> protocol=<negotiated-version> # A negotiated version older than the requested one means the server capped it # — note that in the report; you are then testing an older protocol than a # current client would use. mcp_init() { local url="$1" [ -z "$url" ] && { echo "usage: mcp_init <url>" >&2; return 1; } local want="${MCP_FIELD_TEST_PROTOCOL:-2025-11-25}" local hdr; hdr=$(mktemp) local body_file; body_file=$(mktemp) local code code=$(curl -sS -D "$hdr" -o "$body_file" -w '%{http_code}' -X POST "$url" \ -H "Content-Type: application/json" \ -H "Accept: application/json, text/event-stream" \ -d "{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"initialize\",\"params\":{\"protocolVersion\":\"$want\",\"capabilities\":{},\"clientInfo\":{\"name\":\"field-test\",\"version\":\"1.0.0\"}}}") local sid; sid=$(grep -i '^mcp-session-id:' "$hdr" | awk '{print $2}' | tr -d '\r\n') if [ -z "$sid" ]; then echo "init failed — HTTP $code, no Mcp-Session-Id header returned" >&2 echo "--- response body ---" >&2 cat "$body_file" >&2 echo "--- response headers ---" >&2 cat "$hdr" >&2 rm -f "$hdr" "$body_file" return 1 fi local got; got=$(sed -n 's/^data: //p' "$body_file" | grep -o '"protocolVersion":"[^"]*"' | head -1 | cut -d'"' -f4) [ -z "$got" ] && got=$(grep -o '"protocolVersion":"[^"]*"' "$body_file" | head -1 | cut -d'"' -f4) curl -sS -X POST "$url" \ -H "Content-Type: application/json" \ -H "Accept: application/json, text/event-stream" \ -H "Mcp-Session-Id: $sid" \ -d '{"jsonrpc":"2.0","method":"notifications/initialized"}' >/dev/null rm -f "$hdr" "$body_file" echo "ready sid=$sid protocol=${got:-unknown} requested=$want (HTTP $code)" } # Usage: mcp_call <url> <sid> <method> [JSON_PARAMS] # Prints the JSON-RPC response. SSE framing is stripped when present, and only # the reply is emitted (a single POST can also carry progress notifications, so # emitting every event would break `| jq .result`). A transport failure or an # HTTP >= 400 prints the details and returns non-zero — it never returns 0 with # empty output. Pipe to `jq`. mcp_call() { local url="$1"; local sid="$2"; local method="$3"; local params="${4:-}" [ -z "$url" ] || [ -z "$sid" ] || [ -z "$method" ] && { echo "usage: mcp_call <url> <sid> <method> [params]" >&2; return 1; } local body if [ -z "$params" ]; then body=$(printf '{"jsonrpc":"2.0","id":%d,"method":"%s"}' "$RANDOM" "$method") else body=$(printf '{"jsonrpc":"2.0","id":%d,"method":"%s","params":%s}' "$RANDOM" "$method" "$params") fi local resp_file; resp_file=$(mktemp) local code curl_rc code=$(curl -sS -o "$resp_file" -w '%{http_code}' -X POST "$url" \ -H "Content-Type: application/json" \ -H "Accept: application/json, text/event-stream" \ -H "Mcp-Session-Id: $sid" \ -d "$body") curl_rc=$? if [ "$curl_rc" -ne 0 ] || [ -z "$code" ] || [ "$code" = "000" ]; then echo "TRANSPORT FAILURE calling $method — curl exit $curl_rc, http_code '${code:-none}'." >&2 echo "Server not reachable at $url (check it is still running: mcp_log <log>)." >&2 rm -f "$resp_file" return 1 fi if [ "$code" -ge 400 ]; then echo "HTTP $code from $method — response:" >&2 cat "$resp_file" >&2 rm -f "$resp_file" return 1 fi local sse; sse=$(sed -n 's/^data: //p' "$resp_file") if [ -n "$sse" ]; then local reply; reply=$(printf '%s\n' "$sse" | grep -E '"(result|error)"') printf '%s\n' "${reply:-$sse}" else cat "$resp_file" fi rm -f "$resp_file" } # Usage: mcp_log <server-log-path> [N] (default: 50 lines) # Tail the per-server log printed by mcp_start. Useful when a call surprises # you — pino startup banner, definition lint diagnostics, request handler # errors, upstream calls, and rate-limit warnings all land here. mcp_log() { local log="$1"; local n="${2:-50}" [ -z "$log" ] && { echo "usage: mcp_log <log-path> [n]" >&2; return 1; } tail -n "$n" "$log" } # Usage: mcp_stop <pid> [server-log-path] [port] # Kills the background server and the `bun run` child that actually holds the # port (SIGKILL is not forwarded, so the child must be signalled directly or it # survives as an orphaned listener). Pass the port from mcp_start to have the # stop confirmed against the socket rather than against the wrapper PID. # Removes the server log if a path is given. mcp_stop() { local pid="$1"; local log="${2:-}"; local port="${3:-}" [ -z "$pid" ] && { echo "usage: mcp_stop <pid> [log-path] [port]" >&2; return 1; } local kids; kids=$(pgrep -P "$pid" 2>/dev/null) kill "$pid" $kids 2>/dev/null for _ in $(seq 1 12); do kill -0 "$pid" 2>/dev/null || break sleep 0.25 done if kill -0 "$pid" 2>/dev/null; then echo "PID $pid didn't exit on SIGTERM — sending SIGKILL" kill -9 "$pid" $kids 2>/dev/null sleep 0.5 fi local held="" [ -n "$port" ] && held=$(lsof -ti tcp:"$port" 2>/dev/null | tr '\n' ' ') if [ -n "$held" ]; then echo "WARNING: port $port still held by PID(s) $held after stopping $pid — kill those before re-running" elif kill -0 "$pid" 2>/dev/null; then echo "WARNING: PID $pid still alive after SIGKILL" else echo "stopped pid=$pid${port:+ (port $port free)}" fi [ -n "$log" ] && rm -f "$log" return 0 } HELPER_EOF . /tmp/<project-name>-field-test-9DJ73-K103L.sh mcp_start /absolute/path/to/server # replace with the target server ``` Capture `pid`, `url`, `port`, `log` from the `mcp_start` output — every later call takes them as positional args. Two agents running concurrently in the same project tree each pick their own ID, so their helper paths, server logs, and call scratch never share a name. **Notes** - `MCP_HTTP_PORT` is a *starting* port — the server auto-increments if taken. Helper parses the real URL from the log (`HTTP transport listening at ...`). - If `bun run rebuild` fails, stop. Don't field-test broken code — fix the build first. - Startup wait defaults to 30s. A server that builds or loads a local index at boot can need more — pass a second arg (`mcp_start /path 90`) rather than reading the timeout as a real startup failure. - `pid` is the `bun run` wrapper; the process that actually holds the port is its child. `mcp_stop` signals both — that's why it takes the port. - If a server is already listening on the project's port (`lsof -i :<port>`), confirm with the user before killing it; it may be their own session. If the user isn't available to confirm, abort the field test and surface the port conflict in your response. ### 2. Initialize the session ```bash