mcp-conformance

name	mcp-conformance
description	Use when running, interpreting, baselining, or wiring the MCP conformance harness for a client, remote HTTP server, authorization flow, or CI pipeline.
version	0.1.0

MCP Conformance

Operating rules

• Server mode is HTTP-only. It requires --url; it cannot launch or test a stdio server. Stdio server testing remains open issue #258.
• Pin @modelcontextprotocol/conformance@0.1.16 for this skill. Do not silently switch to 0.2.0-alpha.
• Start the server, then run the harness against the real endpoint. A test file alone is not end-to-end evidence.
• Keep expected failures empty unless a known upstream issue has a narrow, reviewed waiver.

Compatibility boundary: version 0.1.16 rejects --spec-version 2026-07-28; its valid filters are 2025-03-26, 2025-06-18, 2025-11-25, draft, and extension. Do not present it as a 2026 conformance gate. Use it for supported legacy/draft coverage and separately test the modern request contract.

Choose mode and suite

Mode	Target	Suites
server	A remote HTTP server URL	active (default), all, pending
client	A client command launched by the harness	all, core, extensions, auth, metadata, backcompat, draft, sep-835

authorization is not a CLI mode. Test authorization behavior through a supported client/server scenario. Server mode does not accept the client-only --timeout option.

Server workflow

npx @modelcontextprotocol/conformance@0.1.16 server \
  --url http://localhost:3000/mcp \
  --suite active \
  --spec-version 2025-11-25 \
  --verbose

Use --scenario, --suite, --spec-version, --expected-failures, --output-dir, and --verbose deliberately. --timeout is client-only. Every scenario also reports wire-schema validation or harness errors; distinguish those from the server's functional result.

Expected failures

server:
  - server-stateless:some-check-id
client: []

The top level may contain only server and client, with arrays of strings. A mapping form such as - scenario: check-id is invalid. Do not waive both a scenario and one of its checks.

Exit status is intentionally strict:

Result	Exit
Passing, no baseline entry	0
Failing, baseline entry	0
Failing, no baseline entry	1
Passing, baseline entry	1 — stale waiver

CI

Use the root composite action, modelcontextprotocol/conformance@v0.1.16. Its relevant inputs are mode, url, command, expected-failures, suite, scenario, timeout, verbose, and node-version; the action defaults to Node 22. Keep a zero-waiver baseline when possible.

Reporting

Report the pinned harness version, its supported spec filter, mode, suite/scenario, target URL or client command, exit status, output directory, every unbaselined failure, and every stale waiver. Use scripts/run_conformance.py to select server/client form and summarize emitted checks.json files.

References

• references/versions.md — harness version and verification command
• scripts/run_conformance.py — thin runner and result summary