mcp-runtime-troubleshooting

name	mcp-runtime-troubleshooting
description	Debug MCP Runtime cluster, ingress, registry, Sentinel auth, MCPServer pods, and platform UI failures on Kind or k3s. Use when setup, doctor, e2e, or live traffic fails; when investigating 401/404/ImagePullBackOff/ACME/TLS/registry push errors; or when AGENTS.md points here for the full failure-mode checklist.

MCP Runtime — cluster troubleshooting

When to use

• Live cluster misbehaves after setup, upgrade, or config change
• Operator, gateway, registry, or Sentinel symptoms (not unit-test failures)
• Before re-running full setup, scan the checklist for a targeted fix

First steps

kubectl config current-context
./bin/mcp-runtime status
./bin/mcp-runtime cluster doctor
kubectl get ingress -A
kubectl get pods -A | grep -E 'mcp-|registry|traefik|cert-manager'

For Kind contributor clusters, prefer reusing kind-mcp-runtime — see .codex/skills/qa-cluster-bringup/SKILL.md.

For public k3s / mcpruntime.org deploys, also read .codex/skills/k3s-public-ops/SKILL.md and docs/cluster-readiness.md.

Full checklist

Read reference.md end-to-end before diagnosing (ingress, registry, cert-manager, ImagePullBackOff, UI redirect loops, registry push timeouts, k3s NetworkPolicy, duplicate Traefik, and more). Public TLS/DNS detail: mcp-runtime-platform-public skill.

MCPServer pod / gateway sidecar

When a server is deployed but grants, policy, or analytics look wrong:

SERVER=workspace-assistant-mcp
CONTAINER=workspace-assistant-mcp
NS=mcp-servers

kubectl get mcpservers -n "$NS"
POD="$(kubectl get pods -n "$NS" -l app="$SERVER" -o jsonpath='{.items[0].metadata.name}')"
kubectl describe pod -n "$NS" "$POD"
kubectl logs -n "$NS" "$POD" -c "$CONTAINER"
kubectl logs -n "$NS" "$POD" -c mcp-gateway
./bin/mcp-runtime server policy inspect "$SERVER" --namespace "$NS"
kubectl get mcpaccessgrant,mcpagentsession -n "$NS" -o wide

Sidecar container name is mcp-gateway. Many images are distroless — use logs/describe or:

kubectl debug -it -n "$NS" "pod/$POD" --target="$CONTAINER" --image=busybox:1.36 -- sh

Policy reload: the gateway sidecar polls its policy file; wait a few seconds after applying grants/sessions before concluding session_not_found.

Clean start (keep cluster, wipe workloads)

Destructive to application namespaces. From repo root:

to_delete="$(kubectl api-resources --verbs=delete --namespaced -o name | paste -sd, -)"
[ -n "$to_delete" ] && kubectl delete "$to_delete" --all -A --ignore-not-found --grace-period=0 --force
for r in $(kubectl api-resources --verbs=delete --namespaced=false -o name); do
  kubectl delete "$r" --all --ignore-not-found --grace-period=0 --force || true
done
ns_to_delete="$(kubectl get ns --no-headers | awk '{print $1}' | grep -E -v '^(kube-system|kube-public|kube-node-lease|default)$')"
[ -n "$ns_to_delete" ] && printf '%s\n' "$ns_to_delete" | xargs kubectl delete ns

Then rerun setup or contributor bring-up per qa-cluster-bringup.