mcp-qa

Verify MCP tool behavior end-to-end before committing or creating a PR. Prefer agent-callable paths over browser or inspector workflows. ## 1. Quality Gate ```bash pnpm run tsc && pnpm run lint && pnpm run test ``` Fix any failures before proceeding. ## 2. Stdio CLI This is the primary QA path for tool behavior. Stdio runs the local MCP server against prod Sentry without depending on the Cloudflare worker, local `/mcp` route, or Cloudflare OAuth configuration. The test client launches `sentry mcp` from CLI source (`pnpm --dir packages/cli run cli mcp`), which reads mcp-core's build output, so build first when validating code or tool changes: ```bash pnpm -w run build ``` Check auth first. `sentry mcp` uses the CLI's credentials: ```bash pnpm --filter sentry run cli auth status ``` If you are not logged in, log in once (or pass `--access-token` to the test client): ```bash pnpm --filter sentry run cli auth login ``` First prove startup and auth: ```bash pnpm -w run cli --transport stdio --list-tools pnpm -w run cli --transport stdio "who am I?" ``` Then prove the changed behavior with a realistic prod prompt. Choose a prompt that requires the new or modified tool path, uses real org/project/resource inputs, and asks for enough detail to prove the endpoint response is usable. ```bash pnpm -w run cli --transport stdio \ "<prompt that exercises the changed MCP behavior against prod data>" ``` Passing QA means the local CLI prints `Connected to MCP server (stdio)`, uses the expected MCP tool path, and returns real prod data that demonstrates the behavior. Expect the changed tool name in the transcript; with `EXPOSE_ALL_TOOLS` off, expect `search_sentry_tools` followed by `execute_sentry_tool(name: <changed_tool>)` instead. `--list-tools` alone is not QA. For mutating tools, avoid live prod changes unless there is a disposable resource prepared for the test. Add or run a server-level dispatch test with MSW coverage to prove generated schema exposure, constraint injection, and tool dispatch without changing real Sentry data. For output-format changes, also inspect the raw MCP tool result when possible, not only the LLM's final answer. The final answer can add model-specific text that is not part of the tool response. Review raw tool output against `docs/contributing/tool-responses.md`: it should be user-facing, structured, and free of raw API JSON, internal implementation IDs, empty placeholders, and unrelated instructions. ## 3. Real Agent Clients Use these when validating Claude Code, Codex, or behavior that only reproduces in a real agent client: ```bash pnpm -w run build ``` What this verifies: - The CLI is installed and can see the named MCP server - The provider can run a connectivity prompt against Sentry MCP - The final answer includes the authenticated email from `whoami` For agent-client-specific behavior, replace the default harness prompt with the same realistic prod prompt used for stdio QA. Passing QA requires the same changed-tool transcript evidence, not only `whoami`. Real clients do not give stdio subprocesses a TTY, so log in with `sentry auth login` before running the harness. Build first to avoid stale mcp-core code. ## 4. Cloudflare HTTP Only When Relevant Run this only when changes touch Cloudflare, HTTP transport, `/mcp` routing, OAuth, web UI, or hosted-server compatibility. It is not required for ordinary tool handler changes. Start the dev server in a separate terminal or background process: ```bash pnpm dev ``` Then verify it is reachable: ```bash curl -s -o /dev/null -w "%{http_code}" http://localhost:5173/ ``` If `pnpm dev` fails because local Cloudflare/Wrangler is not configured, note the failure and continue with stdio QA for tool behavior. ```bash pnpm -w run cli "who am I?" pnpm -w run cli "list all available tools" pnpm -w run cli "<same realistic prod prompt used for stdio QA>" pnpm -w run cli --mcp-host=http://localhost:5173/mcp/<org> \ "<same realistic prod prompt used for stdio QA>" pnpm -w run cli --mcp-host=http://localhost:5173/mcp/<org>/<project> \ "<same realistic prod prompt used for stdio QA>" ``` Look for `Connected to MCP server (<resolved MCP URL>)` to confirm HTTP transport, plus the same changed-tool transcript evidence required for stdio QA. Use scoped `/mcp/<org>` or `/mcp/<org>/<project>` URLs when validating routing, OAuth, or resource-scope behavior. Use `--setup repo --server sentry` to test the hosted server instead. ## 5. Source Build Stdio Only When Relevant Use this when specifically checking the built local stdio server rather than dev-time source execution: ```bash pnpm -w run build pnpm --filter sentry run cli auth status pnpm -w run cli --transport stdio "who am I?" ``` Look for `Connected to MCP server (stdio)` to confirm stdio transport.