| name | mcp-contract-drift-triage |
| description | Triage MCP test drift by separating stale assertions from real surface regressions after a partial metadata/tool refactor. |
| domain | mcp-review |
| confidence | high |
| source | earned |
When to use
Use this when MCP integration tests start failing after metadata, limits, or tool-surface changes and it is unclear whether the problem is the test, the contract, or a bad merge.
Checklist
- Build first. If the MCP project does not compile, do not treat assertion failures as the top-level issue yet.
- Find the last known green MCP commit and rerun the MCP-focused slice there.
- Diff current MCP files against that green commit, especially:
- options/config models
- metadata documents
- resource URIs
- tool registrations
- request-policy clamp logic
- Classify each failing assertion explicitly:
- stale test expectation: same contract, new value
- response-shape mismatch: renamed/removed JSON field or resource shape
- real regression: shipped tool/metadata behavior no longer coherent
- Watch for partial removals: if a tool disappears from
Tools/* but its limits or metadata remain elsewhere, treat that as hidden contract drift.
- Gate approval on end-to-end coherence, not on a single passing test file.
Proven pattern in this repo
- MCP milestone commit
9cb62c8 was a reliable green baseline.
- A later broken tree mixed old and new MCP contracts in
BaseballMcpOptions, metadata models/services, and Hall of Fame reads.
- The right review move was to prove the old slice still passed, then classify the new failures as stale expectations vs surface-shape drift instead of assuming a pure product regression.
Validation
dotnet build baseball-history.sln --nologodotnet test baseball-history-tests --no-build --nologo --filter Mcp --logger "console;verbosity=minimal"- Re-run the same MCP slice on the last known green MCP commit for comparison when drift classification is unclear.
