field-test

## Context Unit tests (`add-test` skill) verify handler logic with mocked context. Field testing exercises the real HTTP transport with real JSON-RPC: starts the server, calls `initialize`, surfaces the catalog, runs inputs, and checks what a client actually sees. It catches what unit tests miss — awkward input shapes, unhelpful errors, missing format output, drift between `structuredContent` and `content[]`, edge-case surprises. **Actively call the tools. Don't read code and guess.** ### Transport coverage This skill drives an HTTP server because curl + JSON-RPC is the most reliable harness for shell-based agents. The same handlers run on both transports — only the framing differs — so HTTP exercises the full functional surface. Both HTTP session modes are covered: a durable `Mcp-Session-Id` session, and the sessionless initialization a `MCP_SESSION_MODE=stateless` server performs. **Stdio coverage is a boot check only — run this before Step 1.** Run `bun run rebuild && bun run start:stdio < /dev/null`, and confirm the startup logs look clean (banner, expected tool/resource counts, no errors/warnings, no missing-config gripes). Redirecting stdin is what ends the run: the server treats EOF as a shutdown signal, boots fully, then exits on its own, so the log also shows the graceful-shutdown path. Do not background it and reach for `pkill` — a pattern like `pkill -f dist/index.js` matches every other stdio MCP server on the machine, including the ones the calling agent's own session is connected to. Pino logs go to stderr in stdio mode (stdout is reserved for JSON-RPC), so they print straight to the terminal when you run interactively. No need to call tools over stdio — the HTTP pass already covered handler behavior. --- ## Steps ### 1. Start the server Generate a 10-character alphanumeric ID (e.g. `9DJ73-K103L`) and write the helper to `/tmp/<project-name>-field-test-<ID>.sh`. Use that exact path in every subsequent Bash call. **Two agents in the same project tree must pick different IDs** — that's what keeps their helper files, server logs, and call scratch from colliding. The helper itself is **stateless** — every function takes the IDs it needs (server `pid`, `url`, `port`, MCP `sid`, server log path) as positional args. `mcp_start` prints them; the agent threads them through every later call. No env vars, no shared state files. ```bash # Pick your ID — example below uses 9DJ73-K103L. Substitute your own. # (Helper path also encodes the project name so /tmp/ stays grep-friendly.) cat > /tmp/<project-name>-field-test-9DJ73-K103L.sh <<'HELPER_EOF' #!/bin/bash # Field-test helper: stateless wrappers around an MCP HTTP server + JSON-RPC # session. Every function takes the IDs it needs as positional args — the agent # threads pid/url/port/sid/log through each call rather than relying on a state # file or env vars (the Bash tool wipes shell state between calls, and a # pointer file would race the same way two agents race on shared state). # See https://github.com/cyanheads/mcp-ts-core/issues/90, #144. # # Surfaces failures aggressively — field test is for finding things that fail, # so the helper auto-tails logs and prints HTTP status/body on errors instead # of swallowing them. It also measures: every mcp_call prints a one-line # size/latency reading on stderr, and mcp_catalog_size weighs tools/list. # Usage: mcp_start /path/to/server [startup-timeout-seconds] (default: 30) # Builds, starts the HTTP server in the background, waits for the listen line, # and prints: ready pid=<n> url=<u> port=<n> log=<path> # Capture these — every later helper takes them as args. Raise the timeout for # servers that build a local index at boot. mcp_start() { local dir="${1:-$PWD}" local timeout="${2:-30}" local build_log; build_log=$(mktemp /tmp/mcp-field-test-build.XXXXXX) echo "building $dir ..." >&2 if ! (cd "$dir" && bun run rebuild) >"$build_log" 2>&1; then echo "BUILD FAILED — last 30 lines of $build_log:" >&2 tail -30 "$build_log" >&2 return 1 fi rm -f "$build_log" local server_log; server_log=$(mktemp /tmp/mcp-field-test-server.XXXXXX) echo "starting server ..." >&2 (cd "$dir" && bun run start:http) >"$server_log" 2>&1 & local pid=$! local line="" local waited=0 while [ "$waited" -lt "$((timeout * 4))" ]; do line=$(grep -Eo 'listening at http://[^" ]+/mcp' "$server_log" | head -1) [ -n "$line" ] && break if ! kill -0 "$pid" 2>/dev/null; then echo "server exited during startup — last 30 lines of $server_log:" >&2 tail -30 "$server_log" >&2 rm -f "$server_log" return 1 fi sleep 0.25 waited=$((waited + 1)) done if [ -z "$line" ]; then echo "server failed to start within ${timeout}s — last 30 lines of $server_log:" >&2 tail -30 "$server_log" >&2 kill "$pid" 2>/dev/null rm -f "$server_log" return 1 fi local url="${line#listening at }" local port; port=$(echo "$url" | sed -E 's|.*:([0-9]+)/.*|\1|') echo "ready pid=$pid url=$url port=$port log=$server_log" } # Internal: report a failed initialize with the raw exchange, then clean up. _mcp_init_fail() { local msg="$1"; local body_file="$2"; local hdr="$3" echo "init failed — $msg" >&2 echo "--- response body ---" >&2 if [ -s "$body_file" ]; then cat "$body_file" >&2; else echo "(empty)" >&2; fi echo "--- response headers ---" >&2 if [ -s "$hdr" ]; then cat "$hdr" >&2; else echo "(none)" >&2; fi rm -f "$hdr" "$body_file" return 1 } # Usage: mcp_init <url> # Runs `initialize`, sends `notifications/initialized`, prints: # ready sid=<id-or-empty> protocol=<negotiated-version> requested=<want> instructions=<bytes>B (HTTP <code>) # `instructions=` is the byte size of the server's `instructions` string — it # loads into every client session alongside tools/list, so it is the other half # of the per-session context tax mcp_catalog_size weighs. # The initialize *result* is what decides success — a session ID is optional. # A server started with MCP_SESSION_MODE=stateless mints none, and the session # header is then omitted from every later request. Capture BOTH `sid` and # `protocol`: mcp_call takes the protocol as its 5th arg, which is what carries # the negotiated revision when there is no session to carry it. # A negotiated version older than the requested one means the server capped it # — note that in the report; you are then testing an older protocol than a # current client would use. mcp_init() { local url="$1" [ -z "$url" ] && { echo "usage: mcp_init <url>" >&2; return 1; } local want="${MCP_FIELD_TEST_PROTOCOL:-2025-11-25}" local hdr; hdr=$(mktemp) local body_file; body_file=$(mktemp) local code curl_rc code=$(curl -sS -D "$hdr" -o "$body_file" -w '%{http_code}' -X POST "$url" \ -H "Content-Type: application/json" \ -H "Accept: application/json, text/event-stream" \ -d "{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"initialize\",\"params\":{\"protocolVersion\":\"$want\",\"capabilities\":{},\"clientInfo\":{\"name\":\"field-test\",\"version\":\"1.0.0\"}}}") curl_rc=$? if [ "$curl_rc" -ne 0 ] || [ -z "$code" ] || [ "$code" = "000" ]; then _mcp_init_fail "transport failure — curl exit $curl_rc, http_code '${code:-none}'; nothing listening at $url" "$body_file" "$hdr" return 1 fi [ "$code" -ge 400 ] && { _mcp_init_fail "HTTP $code" "$body_file" "$hdr"; return 1; } # Unwrap SSE framing when present; a plain JSON body is used as-is. local payload; payload=$(sed -n 's/^data: //p' "$body_file") [ -z "$payload" ] && payload=$(cat "$body_file") # Pick the reply frame by structure, not by substring: a server that logs to the # client emits `notifications/message` frames first, and a `"level":"error"` or a # log string containing `result` matches a text grep and gets read as the reply. local reply; reply=$(printf '%s\n' "$payload" | jq -c 'select(type=="object" and (has("result") or has("error")))' 2>/dev/null | tail -1) [ -z "$reply" ] && reply="$payload" if printf '%s' "$reply" | grep -q '"error"'; then _mcp_init_fail "server returned a JSON-RPC error" "$body_file" "$hdr" return 1 fi if ! printf '%s' "$reply" | grep -q '"result"'; then _mcp_init_fail "HTTP $code but no JSON-RPC result in the body" "$body_file" "$hdr" return 1 fi local got; got=$(printf '%s' "$reply" | grep -o '"protocolVersion":"[^"]*"' | head -1 | cut -d'"' -f4) if [ -z "$got" ]; then _mcp_init_fail "initialize result declares no protocolVersion" "$body_file" "$hdr" return 1 fi local instr; instr=$(printf '%s' "$reply" | jq -r '.result.instructions // "" | utf8bytelength' 2>/dev/null || echo 0) local sid; sid=$(grep -i '^mcp-session-id:' "$hdr" | awk '{print $2}' | tr -d '\r\n') local init_headers=(-H "Content-Type: application/json" -H "Accept: application/json, text/event-stream" -H "MCP-Protocol-Version: $got") [ -n "$sid" ] && init_headers+=(-H "Mcp-Session-Id: $sid") curl -sS -X POST "$url" "${init_headers[@]}" \ -d '{"jsonrpc":"2.0","method":"notifications/initialized"}' >/dev/null rm -f "$hdr" "$body_file" echo "ready sid=$sid protocol=$got requested=$want instructions=${instr}B (HTTP $code)" } # Internal: one stderr line per call — reply bytes, the content/structured # split, a token estimate, wall-clock. Bytes are the reply as delivered (SSE # framing stripped). `content` is every text block's bytes, `structured` is # structuredContent serialized. The token figure is bytes/4 — an estimate, not # a tokenizer. Wall-clock is curl's time_total for the whole exchange. _mcp_measure() { local method="$1"; local params="$2"; local reply="$3"; local code="$4"; local secs="$5" local total; total=$(printf '%s' "$reply" | wc -c | tr -d ' ') local ms; ms=$(awk -v s="$secs" 'BEGIN { printf "%d", s * 1000 }') local label="$method" local split="" case "$method" in tools/call) local name; name=$(printf '%s' "$params" | jq -r '.name // empty' 2>/dev/null) [ -n "$name" ] && label="$method $name" split=$(printf '%s' "$reply" | jq -r ' (.result // {}) as $r | ([$r.content[]? | select(.type == "text") | .text] | join("") | utf8bytelength) as $c | (if $r.structuredContent == null then "none" else ($r.structuredContent | tojson | utf8bytelength | tostring) end) as $s | "content \($c) · structured \($s)"' 2>/dev/null) ;; resources/read) split=$(printf '%s' "$reply" | jq -r ' "text \([.result.contents[]? | .text // ""] | join("") | utf8bytelength)"' 2>/dev/null) ;; esac local tok; tok=$(awk -v b="$total" 'BEGIN { if (b >= 1000) printf "~%.1fk", b / 4000; else printf "~%d", b / 4 }') echo "⏱ $label · HTTP $code · ${total} B${split:+ ($split)} · $tok tok · ${ms} ms" >&2 } # Usage: mcp_call <url> <sid> <method> [JSON_PARAMS] [protocol] # Prints the JSON-RPC response. SSE framing is stripped when present, and only # the reply is emitted (a single POST can also carry progress notifications, so # emitting every event would break `| jq .result`). A transport failure or an # HTTP >= 400 prints the details and returns non-zero — it never returns 0 with # empty output. Pipe to `jq`. # Every call also prints one measurement line on stderr, e.g. # ⏱ tools/call gbif_search_species · HTTP 200 · 18412 B (content 9100 · structured 8900) · ~4.6k tok · 812 ms # Read it on every call — it is the size/latency evidence the report cites. # `sid` may be empty ('') for a stateless server; the session header is then # omitted. Pass the `protocol` mcp_init printed as the 5th arg — with no # session carrying the negotiation, MCP-Protocol-Version is what tells the # server which revision the request speaks. mcp_call() {